Script optimalisation

Hi,

I have a script that basically interegates a large file (833594152), extracts data and creates an number of files with data that fits within certain criteria. An example of the scripts is below:

for number in `cat linenum`

do

    if [ $counter = 1 ]

         then NO=$number
              counter=`expr $counter + 1`

    elif [ $counter != 1 ]

         then marker=$number
              end=`expr $number - 1`

              filename=exfile$counter

              awk -v NO=$NO -v end=$end 'NR == NO, NR == end' $file1 > $filename

              filename2=compfile$counter

              grep "WORD1 WORD2" $filename > $filename2

                   if [ ! -s $filename2 ]

                       then rm $filename
                            rm $filename2

                       else rm $filename2

                   fi

              NO=$marker
              counter=`expr $counter + 1`
    fi
done

The problem is that it takes a long time to run (i'm estimating atleast 6 years, probably more!!), I know that it is quite "clunky" scripting, but does anyone know how I can possibly streamline this. It seems that the AWK statement is the biggest user of CPU both in time and percentage.

I'm presuming it's the file size that's the problem, but any help would be appreciated.

Thanks

Chris[COLOR="\#738fbf"]

---------- Post updated at 03:52 PM ---------- Previous update was at 03:31 PM ----------

Please ignore the spelling

You know, if you told us what the script is supposed to do it would help.

One place to save time is to lose the expr's and use shell arithmetic operators -

number=$(( $number + 1 ))
# or
number=$(( $somevar - 1 ))

are far more efficient. each execution of awk & grep eat cpu and time as well.

To increase chance you might want to edit your post and insert CODE tags, this little #-button up there in the edit bar when you write/edit a post. It will enhance readability of your code, data or logs and preserve formatting like indention etc.. Please have a look into this, ty.

Ok well, the first 2 lines give a description (albeit vague) of what the script is meant to do. But basically it locates the start and end of non-fixed length data messages within a large file, puts the data in a new file on it's own, checks if a certain detail is present and then either keeps the data message in the file or removes it.

I believe it is the awk statement that is taking the most time as it will have to read in the whole file everytime.

I'll look into the tags and update the post.