Improve performance of echo |awk

chetan.c · April 12, 2012, 9:26am

Hi,

I have a script which looks like this.
Input file
data1^20
data2^30

 
#!/bin/sh
file"/home/Test.txt"
while read line
do
echo $line |awk 'BEGIN { FS = "^" } ; { print $2 }'
echo $line |awk 'BEGIN { FS = "^" } ; { print $1 }' | gzip | wc -c
done <"$file"

How can i improve the performance or better the code?

Thanks,
Chetan.C

Franklin52 · April 12, 2012, 10:02am

Try this:

awk -F"^" '{print $2; print $1 " | gzip | wc -c"}' $file

chetan.c · April 12, 2012, 10:44am

Hi,

Thanks for response.
Actually i need to have the output in the below format only.
data1
20
data2
30.

As i am feeding it to datstage code.

Tried out the code you gave its printing gzip |wc -c also.

Thanks,
Chetan.C

Corona688 · April 12, 2012, 11:08am

Beyond a certain point you can't improve the performance of running gzip thousands of separate times to process tiny strings. This might be some improvement since awk is replaced by pure shell builtins:

while IFS="^" read FIELD1 FIELD2
do
        echo "$FIELD2"
        echo "$FIELD1" | gzip | wc -c
done <"$file"

chetan.c · April 12, 2012, 11:21am

corona688:

Beyond a certain point you can't improve the performance of running gzip thousands of separate times to process tiny strings. This might be some improvement since awk is replaced by pure shell builtins:
while IFS="^" read FIELD1 FIELD2
do
   echo "$FIELD2"
   echo "$FIELD1" | gzip | wc -c
done <"$file"

Works Astoundingly!!!
Thanks a lot.
close to 50% execution time is reduced.

Thanks again,
Chetan.C