help - exec time too long

Shell Programming and Scripting
1

Dear everyone...

thanks to this forum I am able to do everyday more and more complex scripts...but now I come up with problem with optimisation..

problem 1 - optimise:
here is my code:

        while read number
                        do
                             nawk -F "|" -v ms="$number" ' {  if ($2==ms) { print $0 ; } } ' AllnumbersFile | nawk -F "|" '{print $2"|"$14 }'  | nawk -F"|" '{print $1"|"substr($0,13,4)""substr($0,18,2)""substr($0,21,2)}' >> MergedOutput
        done < $dir/1000NumbersFile

and files:

head 1000NumbersFile :

32762879208
48762880032
48762880052
48763881141
48762882490
32761884631
42761884632

head MergedOutput :

38762879208|20090928
32762879208|20090928
48762880032|20090928
48762880052|20090928
48763881141|20090928
48762882490|20090928
32761884631|20090928
42761884632|20090928

AllnumbersFile :

1|38762879208|0|1000|2000-01-01 00:00:00.0|90|1|0|false|0|0|0|0|1070-00-01 01:00:00.0|0|0|1900-01-01 01:00:00.0|1
1|48762880052|0|1000|2000-01-01 00:00:00.0|90|1|0|false|0|0|0|0|1070-00-01 01:00:00.0|0|0|1900-01-01 01:00:00.0|1
1|42761884632|0|1000|2000-01-01 00:00:00.0|90|1|0|false|0|0|0|0|1070-00-01 01:00:00.0|0|0|1900-01-01 01:00:00.0|1
1|48763881141|0|1000|2000-01-01 00:00:00.0|90|1|0|false|0|0|0|0|1070-00-01 01:00:00.0|0|0|1900-01-01 01:00:00.0|1

time is tooo long..

problem 2:
also is it possible taht when system is too busy my script from cron is being stopped. (I notices it becouse it executes everyday from cron and one day additional files is being erased - as script say and the other those files stay - like the end of script is not executed ?!?!?)

---------- Post updated at 01:55 PM ---------- Previous update was at 11:48 AM ----------

anyone??

2

Bumping up posts or double posting is not permitted in these forums.

Please read the rules, which you agreed to when you registered, if you have not already done so.

You may receive an infraction for this. If so, don't worry, just try to follow the rules more carefully. The infraction will expire in the near future

Thank You.

The UNIX and Linux Forums.

3 
nawk -F='|' ' FILENAME=="1000NumbersFile" {[arr[$0]++}
         FILENAME=="AllNumbersFile"
         {
           if($2 in arr)
           {
             tmp= $2 "|" $14
             print $2 "|" substr(tmp,13,4)  substr(tmp,18,2)  substr(tmp,21,2)
           }
         }                
        '  100NumberFiles  AllNumbersFile  >> MergedOutput

You get to tweak this to get what is needed. Some of your code makes no sense to me. The idea is to read all of the first file, then read thru the second file just one time processing matching records. Note: a space between variables concatenates them in awk. I think nawk is new enough to do that correctly.

4

You can try this solution.

awk -F'|' 'NR==FNR{arr[$0]++}$2 in arr{gsub("-","",$14);print $2 FS substr($14,1,8)}' 1000NumbersFile AllNumbersFile >> MergedOutput
5

thanks guys....danmero I use your code..it is awsame..but some explanation would be nice..hehehehe

jim thanks but I do have problem :

nawk: syntax error at source line 1
context is
FILENAME=="1000NumbersFile" >>> {[ <<<
nawk: illegal statement at source line 1
missing ]

what might me a problem

6

If you can tell me what part is not clear maybe I will try to explain.
Your solution was over complicated :rolleyes:

7

well it is little bit hard to undestand which file U access first...becouse both files are just listed after awk.

I do not get this part:

arr[$0]++}$2

:frowning:

8 
awk -F'|' 'NR==FNR{arr[$0]++}$2 in arr{gsub("-","",$14);print $2 FS substr($14,1,8)}' 1000NumbersFile AllNumbersFile >> MergedOutput
9

hey danmero...listen this:

I save exactly 2.5 hours with your solution of execution with this awk!!!

can not belive this :)))