command utilizing high CPU

azmanw2004 · June 7, 2005, 12:19am

Can anybody please help me on how to optimize following command as it use up a lot of CPU :

tail -f $DIR3$DATE4.log |\
while read line
do echo $line | egrep "Processing incoming SMS message" | sed 's/\,/ /g' \
| awk '{print $2}' >> $DIR2/LIST1.$DATE4.log && echo $line | egrep "Processing incoming SMS message" \
| sed 's/\,/ /g' \
| awk '{print $1, $2, $6, $7, $8}' >> $DIR2/DATA1.$DATE4.log && echo $line | egrep "Send_Deliver_Ack: - end" \
| sed 's/\,/ /g' | sed 's/\/ /g' | awk '{print $1, $2, $6, $7}' >> $DIR2/DATA1.$DATE4.log && echo $line \
| egrep "Get_SMS_Data_From_Process_Block: Start" | sed 's/\,/ /g' | sed 's/\/ /g' \
| awk '{print $1, $2, $6, $11}' >> $DIR2/DATA1.$DATE4.log && echo $line | egrep "end completed successfully" \
| sed 's/\,/ /g' | awk '{print $1, $2, $8, $9}' >> $DIR2/DATA1.$DATE4.log

      done &

}

reborg · June 7, 2005, 1:06am

Well, for a start get rid of all the seds and the egrep, since they are only changing the field seperators and searching, both of which can be done by awk, and it is writing everythig to the same ouput file.

That yields this...

tail -f $DIR3$DATE4.log |\
          while read line
          do echo $line |awk -F"[, ] '/Processing incoming SMS message/{print $2 ;print $1, $2, $6, $7, $8}' >> $DIR2/DATA1.$DATE4.log  && \
echo $line | awk -F"[ ,_]" '/Send_Deliver_Ack: - end/{print $1, $2, $6, $7}' >> $DIR2/DATA1.$DATE4.log && \
echo $line | awk -F"[ ,_]" '/Get_SMS_Data_From_Process_Block: Start/ {print $1, $2, $6, $11}' >> $DIR2/DATA1.$DATE4.log && \
echo $line | awk -F"[ ,]" '/end completed successfully/{print $1, $2, $8, $9}' >> $DIR2/DATA1.$DATE4.log

          done &
}

Now it becomes clear that all the processing can be done with awk, but there are still unecessary external calls, Which of course leads to the obvious conclusion that the wrong tools were being used, this should have been an awk script from the start.

like this:

#!/usr/bin/awk -f
#
# filter.awk
#
BEGIN {FS = "[ ,_]"; }


/Processing incoming SMS message/ {
  print $2;
  print $1, $2, $6, $7, $8;
}

/Send_Deliver_Ack: - end/ {
  print $1, $2, $6, $7;
}

/Get_SMS_Data_From_Process_Block: Start/ {
  print $1, $2, $6, $11;
}

/end completed successfully/ {
  print $1, $2, $8, $9;
}

and the command then becomes:
tail -f $DIR3$DATE4.log | filter.awk > DATA1.$DATE4.log

azmanw2004 · June 7, 2005, 2:03am

But I need the sed as the whole script is doing calculation base on hour, minutes, second and ms.

reborg · June 7, 2005, 2:05am

It is not needed anywhere in what you posted, what I posted merely replaces what you posted.

azmanw2004 · June 7, 2005, 4:34am

Could anybody help me to optimize the script below:

#!/bin/ksh
DATE1=`date '+%d%m%y'`
DATE2=`date '+%e/%m/%Y'`
DATE3=`date '+%H:%M:%S'`
DATE4=`date '+%Y%m%d'`
DATE5=`date '+%d'`

DIR1=/app/application/log/data/dir1/
DIR2=/app/application/log/data/dir2/
DIR3=/app/application/log/data/dir3/
DIR4=/app/application/log/data/dir4/
DIR5=/app/application/log/data/dir5/

Process1() {

tail -f $DIR3$DATE4.log |\
while read line
do echo $line | egrep "Processing incoming SMS message" | sed 's/\,/ /g' \
| awk '{print $2}' >> $DIR2/LIST1.$DATE4.log && echo $line | egrep "Processing incoming SMS message" \
| sed 's/\,/ /g' \
| awk '{print $1, $2, $6, $7, $8}' >> $DIR2/DATA1.$DATE4.log && echo $line | egrep "Send_DATA3ver_Ack: - end" \
| sed 's/\,/ /g' | sed 's/\/ /g' | awk '{print $1, $2, $6, $7}' >> $DIR2/DATA1.$DATE4.log && echo $line \
| egrep "Get_SMS_Data_From_Process_Block: Start" | sed 's/\,/ /g' | sed 's/\/ /g' \
| awk '{print $1, $2, $6, $11}' >> $DIR2/DATA1.$DATE4.log && echo $line | egrep "end completed successfully" \
| sed 's/\,/ /g' | awk '{print $1, $2, $8, $9}' >> $DIR2/DATA1.$DATE4.log

      done &

}

Process2() {

tail -f $DIR1$DATE4.log |\
while read line
do echo $DATA2 | egrep "Process_process_blocks: Start" | sed 's/\,/ /g' \
| awk '{print $1, $2, $5, $6}' >> $DIR2/DATA1.$DATE4.log && echo $DATA2 | egrep "Process_process_blocks: End" \
| sed 's/\,/ /g' | awk '{print $1, $2, $5, $6}' >> $DIR2/DATA1.$DATE4.log
done &
}

LIST() {

while read i
do

DATALOG1=`cat $DIR2/DATA1.$DATE4.log | grep $i | grep "Processing incoming SMS" | awk '{print $1, $2}'`
DATALOG2=`cat $DIR2/DATA1.$DATE4.log | grep $i | grep "DATA3ver Ack:" | awk '{print $1}'`
DATALOG3=`cat $DIR2/DATA1.$DATE4.log | grep $i | grep "Process_process_blocks: Start" | awk '{print $1}'`
DATALOG4=`cat $DIR2/DATA1.$DATE4.log | grep $i | grep "Process_process_blocks: End" | awk '{print $1}'`
DATALOG5=`cat $DIR2/DATA1.$DATE4.log | grep $i | grep "SMS Start" | awk '{print $1}'`
DATALOG6=`cat $DIR2/DATA1.$DATE4.log | grep $i | grep "completed successfully" | sed '$!d' | awk '{print $1}'`

if [ $DATALOG3 -n str ]
then
echo ""
else

   DATALOG11=\`echo $DATALOG1 | sed 's/\\:/ /g' | sed 's/\\./ /g'\`
   DATALOG21=\`echo $DATALOG2 | sed 's/\\:/ /g' | sed 's/\\./ /g'\`
   DATALOG31=\`echo $DATALOG3 | sed 's/\\:/ /g' | sed 's/\\./ /g'\`
   DATALOG41=\`echo $DATALOG4 | sed 's/\\:/ /g' | sed 's/\\./ /g'\`
   DATALOG51=\`echo $DATALOG5 | sed 's/\\:/ /g' | sed 's/\\./ /g'\`
   DATALOG61=\`echo $DATALOG6 | sed 's/\\:/ /g' | sed 's/\\./ /g'\`

        DATALOG12=\`echo $DATALOG11 | awk '\{print $1\}'\`
        DATALOG22=\`echo $DATALOG21 | awk '\{print $1\}'\`

        DATALOG32=\`echo $DATALOG31 | awk '\{print $1\}'\`
        DATALOG42=\`echo $DATALOG41 | awk '\{print $1\}'\`

        DATALOG52=\`echo $DATALOG51 | awk '\{print $1\}'\`
        DATALOG62=\`echo $DATALOG61 | awk '\{print $1\}'\`

             if [ $DATALOG12 = "00" ] && [ $DATALOG12 = "01" ] && [ $DATALOG12 = "02" ] && [ $DATALOG12 = "03" ] && \\
                [ $DATALOG12 = "04" ] && [ $DATALOG12 = "05" ] && [ $DATALOG12 = "06" ] && [ $DATALOG12 = "07" ] && \\ 
                [ $DATALOG12 = "08" ] && [ $DATALOG12 = "09" ]
                then
                   DATALOG12=\`echo $DATALOG12 | sed 's/0//g'\`
             fi
             if [ $DATALOG22 = "00" ] && [ $DATALOG22 = "01" ] && [ $DATALOG22 = "02" ] && [ $DATALOG22 = "03" ] && \\
                [ $DATALOG22 = "04" ] && [ $DATALOG22 = "05" ] && [ $DATALOG22 = "06" ] && [ $DATALOG22 = "07" ] && \\
                [ $DATALOG22 = "08" ] && [ $DATALOG22 = "09" ]
                then
                   DATALOG22=\`echo $DATALOG22 | sed 's/0//g'\`
             fi
             if [ $DATALOG32 = "00" ] && [ $DATALOG32 = "01" ] && [ $DATALOG32 = "02" ] && [ $ =DATALOG32 "03" ] && \\
                [ $DATALOG32 = "04" ] && [ $DATALOG32 = "05" ] && [ $DATALOG32 = "06" ] && [ $ =DATALOG32 "07" ] && \\
                [ $DATALOG32 = "08" ] && [ $DATALOG32 = "09" ]
                then
                   DATALOG32=\`echo $DATALOG32 | sed 's/0//g'\`
             fi
             if [ $DATALOG42 = "00" ] && [ $DATALOG42 = "01" ] && [ $DATALOG42 = "02" ] && [ $DATALOG42 = "03" ] && \\
                [ $DATALOG42 = "04" ] && [ $DATALOG42 = "05" ] && [ $DATALOG42 = "06" ] && [ $DATALOG42 = "07" ] && \\
                [ $DATALOG42 = "08" ] && [ $DATALOG42 = "09" ]
                then
                   DATALOG42=\`echo $DATALOG42 | sed 's/0//g'\`
             fi
             if [ $DATALOG52 = "00" ] && [ $DATALOG52 = "01" ] && [ $DATALOG52 = "02" ] && [ $DATALOG52 = "03" ] && \\
                [ $DATALOG52 = "04" ] && [ $DATALOG52 = "05" ] && [ $DATALOG52 = "06" ] && [ $DATALOG52 = "07" ] && \\
                [ $DATALOG52 = "08" ] && [ $DATALOG52 = "09" ]
                then
                   DATALOG52=\`echo $DATALOG52 | sed 's/0//g'\`
             fi
             if [ $DATALOG62 = "00" ] && [ $DATALOG62 = "01" ] && [ $DATALOG62 = "02" ] && [ $DATALOG62 = "03" ] && \\
                [ $DATALOG62 = "04" ] && [ $DATALOG62 = "05" ] && [ $DATALOG62 = "06" ] && [ $DATALOG62 = "07" ] && \\
                [ $DATALOG62 = "08" ] && [ $DATALOG62 = "09" ]
                then
                   DATALOG62=\`echo $DATALOG62 | sed 's/0//g'\`
             fi
                   HH=\`expr $DATALOG22 - $DATALOG12\`
                   if [ $HH -ne 0 ]
                      then
                         DATA11=\`echo $DATALOG11 | awk '\{print $2\}'\`
                         DATA12=\`echo $DATALOG21 | awk '\{print $2\}'\` 
           	             DATA13=\`expr $HH \\* 60\`
                         DATA14=\`expr $DATA12 \+ $DATA13\`
                         DATA1MM=\`expr $DATA14 - $DATA11\`
                   else
                         DATA11=\`echo $DATALOG11 | awk '\{print $2\}'\`
                         DATA12=\`echo $DATALOG21 | awk '\{print $2\}'\`
                         DATA1MM=\`expr $DATA12 - $DATA11\`
                   fi
                   if [ $DATA1MM -ne 0 ]
                      then
                         DATA11=\`echo $DATALOG11 | awk '\{print $3\}'\`
                         DATA12=\`echo $DATALOG21 | awk '\{print $3\}'\`
                         DATA13=\`expr $DATA1MM \\* 60\`
                         DATA14=\`expr $DATA12 \+ $DATA13\`
                         DATA1SS=\`expr $DATA14 - $DATA11\`
                   else
                         DATA11=\`echo $DATALOG11 | awk '\{print $3\}'\`
                         DATA12=\`echo $DATALOG21 | awk '\{print $3\}'\`
                         DATA1SS=\`expr $DATA12 - $DATA11\`
                   fi
                   if [ $DATA1SS -ne 0 ]
                      then
                         DATA11=\`echo $DATALOG11 | awk '\{print $4\}'\`
                         DATA12=\`echo $DATALOG21 | awk '\{print $4\}'\`
                         DATA13=\`expr $DATA1SS \\* 1000\`
                         DATA14=\`expr $DATA12 \+ $DATA13\`
                         DATA1SSS=\`expr $DATA14 - $DATA11\`
                   else
                         DATA11=\`echo $DATALOG11 | awk '\{print $4\}'\`
                         DATA12=\`echo $DATALOG21 | awk '\{print $4\}'\`
                         DATA1SSS=\`expr $DATA12 - $DATA11\`
                   fi

                   HH=\`expr $DATALOG42 - $DATALOG32\`
                   if [ $HH -ne 0 ]
                      then
                         DATA2=\`echo $DATALOG31 | awk '\{print $2\}'\`
                         DATA22=\`echo $DATALOG41 | awk '\{print $2\}'\`
                         DATA23=\`expr $HH \\* 60\`
                         DATA24=\`expr $DATA22 \+ $DATA23\`
                         DATA2MM=\`expr $DATA24 - $DATA2\`
                   else
                         DATA2=\`echo $DATALOG31 | awk '\{print $2\}'\`
                         DATA22=\`echo $DATALOG41 | awk '\{print $2\}'\`
                         DATA2MM=\`expr $DATA22 - $DATA2\`

                   fi
                   if [ $DATA2MM -ne 0 ]
                      then

                         DATA2=\`echo $DATALOG31 | awk '\{print $3\}'\`
                         DATA22=\`echo $DATALOG41 | awk '\{print $3\}'\`
                         DATA23=\`expr $DATA2MM \\* 60\`
                         DATA24=\`expr $DATA22 \+ $DATA23\`
                         DATA2SS=\`expr $DATA24 - $DATA2\`
                   else
                         DATA2=\`echo $DATALOG31 | awk '\{print $3\}'\`
                         DATA22=\`echo $DATALOG41 | awk '\{print $3\}'\`
                         DATA2SS=\`expr $DATA22 - $DATA2\`

                   fi
                   if [ $DATA2SS -ne 0 ]
                      then
                         DATA2=\`echo $DATALOG31 | awk '\{print $4\}'\`
                         DATA22=\`echo $DATALOG41 | awk '\{print $4\}'\`
                         DATA23=\`expr $DATA2SS \\* 1000\`
                         DATA24=\`expr $DATA22 \+ $DATA23\`
                         DATA2SSS=\`expr $DATA24 - $DATA2\`
                   else
                         DATA2=\`echo $DATALOG31 | awk '\{print $4\}'\`
                         DATA22=\`echo $DATALOG41 | awk '\{print $4\}'\`
                         DATA2SSS=\`expr $DATA22 - $DATA2\`
                   fi 
                   
                   HH=\`expr $DATALOG32 - $DATALOG22\`
                   if [ $HH -ne 0 ]
                      then
                         DATA31=\`echo $DATALOG21 | awk '\{print $2\}'\`
                         DATA32=\`echo $DATALOG31 | awk '\{print $2\}'\`
                         DATA33=\`expr $HH \\* 60\`
                         DATA34=\`expr $DATA32 \+ $DATA33\`
                         DATA3MM=\`expr $DATA34 - $DATA31\`
                   else
                         DATA31=\`echo $DATALOG21 | awk '\{print $2\}'\`
                         DATA32=\`echo $DATALOG31 | awk '\{print $2\}'\`
                         DATA3MM=\`expr $DATA32 - $DATA31\`

                   fi
                   if [ $DATA3MM -ne 0 ]
                      then

                         DATA31=\`echo $DATALOG21 | awk '\{print $3\}'\`
                         DATA32=\`echo $DATALOG31 | awk '\{print $3\}'\`
                         DATA33=\`expr $DATA3MM \\* 60\`
                         DATA34=\`expr $DATA32 \+ $DATA33\`
                         DATA3SS=\`expr $DATA34 - $DATA31\`
                   else
                         DATA31=\`echo $DATALOG21 | awk '\{print $3\}'\`
                         DATA32=\`echo $DATALOG31 | awk '\{print $3\}'\`
                         DATA3SS=\`expr $DATA32 - $DATA31\`

                   fi
                   if [ $DATA3SS -ne 0 ]
                      then
                         DATA31=\`echo $DATALOG21 | awk '\{print $4\}'\`
                         DATA32=\`echo $DATALOG31 | awk '\{print $4\}'\`
                         DATA33=\`expr $DATA3SS \\* 1000\`
                         DATA34=\`expr $DATA32 \+ $DATA33\`
                         DATA3SSS=\`expr $DATA34 - $DATA31\`
                   else
                         DATA31=\`echo $DATALOG21 | awk '\{print $4\}'\`
                         DATA32=\`echo $DATALOG31 | awk '\{print $4\}'\`
                         DATA3SSS=\`expr $DATA32 - $DATA31\`
                   fi
                   HH=\`expr $DATALOG62 - $DATALOG52\`
                   if [ $HH -ne 0 ]
                      then
                         DATA41=\`echo $DATALOG51 | awk '\{print $2\}'\`
                         DATA42=\`echo $DATALOG61 | awk '\{print $2\}'\`
                         DATA43=\`expr $HH \\* 60\`
                         DATA44=\`expr $DATA42 \+ $DATA43\`
                         DATA4MM=\`expr $DATA44 - $DATA41\`
                   else
                         DATA41=\`echo $DATALOG51 | awk '\{print $2\}'\`
                         DATA42=\`echo $DATALOG61 | awk '\{print $2\}'\`
                         DATA4MM=\`expr $DATA42 - $DATA41\`

                   fi
                   if [ $DATA4MM -ne 0 ]
                      then

                         DATA41=\`echo $DATALOG51 | awk '\{print $3\}'\`
                         DATA42=\`echo $DATALOG61 | awk '\{print $3\}'\`
                         DATA43=\`expr $DATA4MM \\* 60\`
                         DATA44=\`expr $DATA42 \+ $DATA43\`
                         DATA4SS=\`expr $4 - $1\`
                   else
                         DATA41=\`echo $DATALOG51 | awk '\{print $3\}'\`
                         DATA42=\`echo $DATALOG61 | awk '\{print $3\}'\`
                         DATA4SS=\`expr $DATA42 - $DATA41\`

                   fi
                   if [ $DATA4SS -ne 0 ]
                      then
                         DATA41=\`echo $DATALOG51 | awk '\{print $4\}'\`
                         DATA42=\`echo $DATALOG61 | awk '\{print $4\}'\`
                         DATA43=\`expr $DATA4SS \\* 1000\`
                         DATA44=\`expr $DATA42 \+ $DATA43\`
                         DATA4SSS=\`expr $DATA44 - $DATA41\`
                   else
                         DATA41=\`echo $DATALOG51 | awk '\{print $4\}'\`
                         DATA42=\`echo $DATALOG61 | awk '\{print $4\}'\`
                         DATA4SSS=\`expr $DATA42 - $DATA41\`
                   fi
                   HH=\`expr $DATALOG52 - $DATALOG42\`
                   if [ $HH -ne 0 ]
                      then
                         DATA51=\`echo $DATALOG41 | awk '\{print $2\}'\`
                         DATA52=\`echo $DATALOG61 | awk '\{print $2\}'\`
                         DATA53=\`expr $HH \\* 60\`
                         DATA54=\`expr $DATA52 \+ $DATA53\`
                         DATA5MM=\`expr $DATA54 - $DATA51\`
                   else
                         DATA51=\`echo $DATALOG41 | awk '\{print $2\}'\`
                         DATA52=\`echo $DATALOG61 | awk '\{print $2\}'\`
                         DATA5MM=\`expr $DATA52 - $DATA51\`

                   fi
                   if [ $DATA5MM -ne 0 ]
                      then

                         DATA51=\`echo $DATALOG41 | awk '\{print $3\}'\`
                         DATA52=\`echo $DATALOG61 | awk '\{print $3\}'\`
                         DATA53=\`expr $DATA5MM \\* 60\`
                         DATA54=\`expr $DATA52 \+ $DATA53\`
                         DATA5SS=\`expr $DATA54 - $DATA51\`
                   else
                         DATA51=\`echo $DATALOG41 | awk '\{print $3\}'\`
                         DATA52=\`echo $DATALOG61 | awk '\{print $3\}'\`
                         DATA5SS=\`expr $DATA52 - $DATA51\`

                   fi
                   if [ $DATA5SS -ne 0 ]
                      then
                         DATA51=\`echo $DATALOG41 | awk '\{print $4\}'\`
                         DATA52=\`echo $DATALOG61 | awk '\{print $4\}'\`
                         DATA53=\`expr $DATA5SS \\* 1000\`
                         DATA54=\`expr $DATA52 \+ $DATA53\`
                         DATA5SSS=\`expr $DATA54 - $DATA51\`
                   else
                         DATA51=\`echo $DATALOG41 | awk '\{print $4\}'\`
                         DATA52=\`echo $DATALOG61 | awk '\{print $4\}'\`
                         DATA5SSS=\`expr $DATA52 - $DATA51\`
                   fi
                   HH=\`expr $DATALOG62 - $DATALOG12\`
                   if [ $HH -ne 0 ]
                      then
                         DATA6=\`echo $DATALOG11 | awk '\{print $2\}'\`
                         DATA62=\`echo $DATALOG61 | awk '\{print $2\}'\`
                         DATA63=\`expr $HH \\* 60\`
                         DATA64=\`expr $DATA62 \+ $DATA63\`
                         DATA6MM=\`expr $DATA64 - $DATA6\`
                   else
                         DATA6=\`echo $DATALOG11 | awk '\{print $2\}'\`
                         DATA62=\`echo $DATALOG61 | awk '\{print $2\}'\`
                         DATA6MM=\`expr $DATA62 - $DATA6\`

                   fi
                   if [ $DATA6MM -ne 0 ]
                      then
                         DATA6=\`echo $DATALOG11 | awk '\{print $3\}'\`
                         DATA62=\`echo $DATALOG61 | awk '\{print $3\}'\`
                         DATA63=\`expr $DATA6MM \\* 60\`
                         DATA64=\`expr $DATA62 \+ $DATA63\`
                         DATA6SS=\`expr $DATA64 - $DATA6\`
                   else
                         DATA6=\`echo $DATALOG11 | awk '\{print $3\}'\`
                         DATA62=\`echo $DATALOG61 | awk '\{print $3\}'\`
                         DATA6SS=\`expr $DATA62 - $DATA6\`

                   fi
                   if [ $DATA6SS -ne 0 ]
                      then
                         DATA6=\`echo $DATALOG11 | awk '\{print $4\}'\`
                         DATA62=\`echo $DATALOG61 | awk '\{print $4\}'\`
                         DATA63=\`expr $DATA6SS \\* 1000\`
                         DATA64=\`expr $DATA62 \+ $DATA63\`
                         DATA6SSS=\`expr $DATA64 - $DATA6\`
                   else
                         DATA6=\`echo $DATALOG11 | awk '\{print $4\}'\`
                         DATA62=\`echo $DATALOG61 | awk '\{print $4\}'\`
                         DATA6SSS=\`expr $DATA62 - $DATA6\`
                   fi
                   

                         echo "Start Time   Trans ID	DATA1	DATA3	DATA2	DATA3	DATA1	End 2 End" &gt;&gt; BDSLA$DATE4.log 
     	             echo "$DATALOG1	$DATA1SSS	$DATA3SSS		$DATA2SSS	$DATA5SSS		$DATA4SSS	$DATA6SSS" &gt;&gt; BDSLA$DATE4.log



 sleep 10

fi
done

}<$DIR2/LIST1.$DATE4.log

############## MAIN ##################

DATA11
sleep 5
DATA2
sleep 60
LIST

######################################

Perderabo · June 7, 2005, 8:45am

azmanw2004, please don't start a second thread in the middle of a question. I have merged your threads.

zazzybob · June 7, 2005, 9:26am

azmanw2004, please see reborgs response - you are given a clear example of how to perform an optimisation on the code that appears in your Process1() function by using a single tool - awk - to do the job. The same optimisation logic can easily be applied to Process2().

Looking at the rest of the code, I'd say the whole thing could be coded as an awk script(s) inside a lightweight shell script wrapper.

Cheers
ZB