Combining awk code into one

Shell Programming and Scripting
1

the following code works perfectly for me:

# AWK 1
gawk -F, '/,'${ThisMonthDOW}' '${ThisMonthMON}' [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dow

## AWK 2
                                                                gawk -F, '/,(Mon|Tue|Wed|Thu|Fri|Sat|Sun) '${ThisMonthMON}' '${ThisMonthDAY}' [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10} 
                                                                                }
                                                                }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dom

However, as you can see, it is inefficient as im running awk on a log file 2 different times when I should be able to run it once.

my question is, how can i combine this into one? so that it takes the different log files as arguments. and then searches for the distinct patterns in each of the log files and then outputs the results to the different text files ending in ".dow" and ".dom".

2

Hi,
Maybe (not tested but permit in gawk) :

gawk -F, '/,'${ThisMonthDOW}' '${ThisMonthMON}' [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dow}  
                                                                                }
                                                                }
/,(Mon|Tue|Wed|Thu|Fri|Sat|Sun) '${ThisMonthMON}' '${ThisMonthDAY}' [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
                                                                        if (NF == 10)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                }
                                                                        else if (NF == 11)
                                                                                {
                                                                                split($10, B,"--");
                                                                                if (B[2] >= B[1])
                                                                                        {print B[2]-B[1] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] == 0))
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if ((B[1] > B[2]) && (B[2] > 0))
                                                                                        {print B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] == B[1])
                                                                                        {print "0" "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                else if (B[2] < B[1])
                                                                                        {print B[1]-B[2] "," $3 "," $10 >> total.entries.'${NTIME}'.mon.dom} 
                                                                                }
                                                                }' ${TagFile}.one.${NMON}.mon

Beware: you must empty your files total.entries*

Regards.

1 Like
3

this doesnt seem to be working. no files get created.

if it helps, the content of the data file has many lines that look like this:

0,records_scms1,1506350978,66,DATA.TXT,800pages,records_scms2,6395,0,66--66,3400
4

Hello SkySmart,

Appreciate that you have shown us what you have tried :b:. Could you please let us know the sample Input_file and expected output too, so that we could see how we could help here.

Thanks,
R. Singh

5

my apologies. the code provided my disedorgue works. there was a mistake on my end. sorry for inconvenience.

6

Rearranging the script a little

gawk -F, '/,'${ThisMonthDOW}' '${ThisMonthMON}' [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
if (NF == 10)   {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if  (B[2] == B[1])                       {print "0"       "," $3 "," $10} 
                  else if  (B[2] < B[1])                        {print B[1]-B[2] "," $3 "," $10} 
        }
else if (NF == 11) {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if (B[2] == B[1])                        {print "0"       "," $3 "," $10} 
                  else if (B[2] < B[1])                         {print B[1]-B[2] "," $3 "," $10} 
        }
 }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dow


 gawk -F, '/,(Mon|Tue|Wed|Thu|Fri|Sat|Sun) '${ThisMonthMON}' '${ThisMonthDAY}' [0-9][0-9]:[0-9][0-9]:[0-9][0-9] '${ThisMonthYEA}',/ {
if (NF == 10)   {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if  (B[2] == B[1])                       {print "0"       "," $3 "," $10} 
                  else if  (B[2] < B[1])                        {print B[1]-B[2] "," $3 "," $10} 
        }
else if (NF == 11) {split($10, B,"--");
                 if        (B[2] >= B[1])                       {print B[2]-B[1] "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] == 0))        {print "0"       "," $3 "," $10} 
                  else if ((B[1] > B[2]) && (B[2] > 0))         {print B[2]      "," $3 "," $10} 
                  else if  (B[2] == B[1])                       {print "0"       "," $3 "," $10} 
                  else if  (B[2] < B[1])                        {print B[1]-B[2] "," $3 "," $10} 
        }
 }' ${TagFile}.one.${NMON}.mon > total.entries.${NTIME}.mon.dom

shows that there's a decent amount of redundancy. There'd be no difference in the logics had we 10 or 11 fields, nor the output file being ".dow" or ".dom" (which in turn only depends on a small difference of the input line. The split is performed 4 times, 3 times too many.
So, in essence, your pages of script could be reduced to a little bit of logics and a few lines of print statements...

Are you aware that your sample line would not match either regex?

1 Like
7

Unable to test as neither variable contents nor data samples are available, but could serve as a pointer in a simplification direction:

awk -F, -vTMW=${ThisMonthDOW} -vTMM=${ThisMonthMON} -vTMD=${ThisMonthDAY} -vTMY=${ThisMonthYEA} -vNTM=${NTIME} '
BEGIN           {FN[1] = "total.entries."NTM".mon.dow"
                 FN[2] = "total.entries."NTM".mon.dom"
                }

                {TYPE = 0
                }
$0 ~ "," TMW " " TMM " [0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] " TMY
                {TYPE = 1
                }
$0 ~ ",(Mon|Tue|Wed|Thu|Fri|Sat|Sun) " TMM " " TMD " [0-9][0-9]:[0-9][0-9]:[0-9][0-9] " TMY
                {TYPE = 2
                }

TYPE && ((NF == 10) || (NF == 11))
                {split($10, B,"--");
                 OUT = abs(B[1] - B[2])
                 if   ((B[1] > B[2]) && (B[2] >= 0))    OUT = B[2]
                 print OUT  "," $3 "," $10 > FN[TYPE]
                } 
 ' ${TagFile}.one.${NMON}.mon
1 Like