awk command issue

Hi All,
I am trying to perform certain task using awk command but the command is not returning expected output though it is running fine independently but not in for loop.

req:

  1. I am searching a pattern (say pattern1) in a dated file
  2. trying to search the last occurrence of the another pattern (say pattern2) before last occurrence of pattern1.

what i am trying do:

  1. search the file name where pattern1 is pattern (for today's date).
  2. print the file from start to the last occurrence of pattern1
  3. search first occurrence of pattern2 from the last occurrence of pattern1.

here is code snippet:

for i in PUAFZ00 AATGY00 AATHB00 AATGZ00; 
do 
echo =======================================================================================================================================;
echo -e " Symbol - $(tput setaf 1) $i ";
for d in Mon Tue Wed Thu Fri; 
do 
ydate=`date +%Y%m%d -d "last $d"`; 
filename=`grep $i /updates/logs/log*$ydate* | cut -d ':' -f1 | uniq`; 
updatefile=$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 |  awk 'NF{NF-=1}1' FS='_' OFS='_' ); yupd=`echo $updatefile"_"$ydate`; 
completiontime=( $(grep $yupd /home/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) ); 
echo  "$(tput sgr 0) Processing_date - $(tput setaf 2) $ydate $(tput sgr 0) UpdatePackage - $(tput setaf 6) $updatefile $(tput sgr 0) Processing_Time $(tput setaf 3)${completiontime[@]:0} $(tput sgr 0)" ;
done;
echo =======================================================================================================================================; done

Unexpected output received on command:

$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 |  awk 'NF{NF-=1}1' FS='_' OFS='_' );

If i pass this command by passing the values on prompt expected output is returned.

Without seeing any sample data, seeing the results that are being produced, and seeing the results that are desired (all in CODE tags); it is very difficult to make any suggestions about what code changes should be made.

On first sight it pops up that within single quotes shell variables will not be expanded, so

$(awk '1;/$i/{exit}' $filename

will not work as expected.

for i in PUAFZ00; 
do 
echo =======================================================================================================================================;
echo -e " Symbol - $(tput setaf 1) $i ";
for d in Mon; 
do 
ydate=`date +%Y%m%d -d "last $d"`; 
filename=`grep $i /home/lim/updates/logs/log*$ydate* | cut -d ':' -f1 | uniq`; 
updatefile=$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 |  awk 'NF{NF-=1}1' FS='_' OFS='_' ); yupd=`echo $updatefile"_"$ydate`; 
completiontime=( $(grep $yupd /home/lim/config/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) ); 
echo  "$(tput sgr 0) Processing_date - $(tput setaf 2) $ydate $(tput sgr 0) UpdatePackage - $(tput setaf 6) $updatefile $(tput sgr 0) Processing_Time $(tput setaf 3)${completiontime[@]:0} $(tput sgr 0)" ;
done;
echo =======================================================================================================================================; done

output:
Symbol - PUAFZ00
Processing_date - 20161017 UpdatePackage - upd_0_k36 Processing_Time 21:25:21 EDT 2016

When i run the awk command indepndently; i get the result: (this is expected)

grep command

grep PUAFZ00 /home/lim/updates/logs/log*`date +%Y%m%d -d "last Mon"`* | cut -d ':' -f1 | uniq

output:
/home/lim/updates/logs/log.0_20161017.old

awk command:

awk '1;/PUAFZ00/{exit}' /home/lim/updates/logs/log.0_20161017.old | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 |  awk 'NF{NF-=1}1' FS='_' OFS='_'

output awk:[this is expected one]
Unpack complete upd_0_pluz

---------- Post updated at 03:09 AM ---------- Previous update was at 02:56 AM ----------

awk -v var="$i" '1;/var/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 |  awk 'NF{NF-=1}1' FS='_' OFS='_' 

is also returning the same output:
UpdatePackage - upd_0_k36

---------- Post updated at 03:43 AM ---------- Previous update was at 03:09 AM ----------

awk -v var=$i -v var2="$filename" '1;/var/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}'

output:

This is one step further, but still not enough. /var/ will look for the string "var" verbatim.
Try

$0 ~ var

.

 awk -v var=$i -v var2="$filename" '1;/var/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 |  awk 'NF{NF-=1}1' FS='_' OFS='_' 
awk -v var=$i -v var2="$filename" '1;/var/{exit}' var2 | awk '/upd_/ {a=$0} END{print a}'

Please help on this:

awk -v var=$i -v var2="$filename" '1;/$0/{exit}' var2 | awk '/upd_/ {a=$0} END{print a}'

Try

awk -v var=$i  '1; $0 ~ var {exit}' $filename

You need to finely differentiate between what is IN the awk command, and what is outside (i.e. shell).

1 Like

Code worked. thanks

Looking at it again, the looong pipe could be reduced to one single awk command, although without decent input sample this is impossible to verify. Try

awk -v var=$i '
/upd_/  {a = $0}                        # keep last line containing "upd_" before the one -
$0 ~ var {exit}                         # containing the pattern in "var"; when found, leave
                                        #      script jumping to END section
END     {split (a, T, "/")              # spread "a" variable into "T" array on "/"
         sub (/\..*$/, "", T[6])        # remove everything after first dot "."
         sub (/_[^_]*$/, "", T[6])      # remove the last part after last "_" in element
         print T[6]                     # print result in element
        }
' $filename

and come back with the results.

EDIT: / added after abhii's comments/errors...

for me

awk -v var=$i  '1; $0 ~ var {exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d ' ' -f3

did work to get the filename with date format (what was required)

though the code you have posted given me the error:

awk -v var=$i '/upd_/  {a = $0} $0 ~ var {exit} END {split (a, T, "/") sub (/\..*$/, "", T[6]) sub (/_[^_]*$, "", T[6]) print T[6] }' $filename
awk: /upd_/  {a = $0} $0 ~ var {exit} END {split (a, T, "/") sub (/\..*$/, "", T[6]) sub (/_[^_]*$, "", T[6]) print T[6] }
awk:                                                                                       ^ unterminated regexp
awk: cmd. line:1: /upd_/  {a = $0} $0 ~ var {exit} END {split (a, T, "/") sub (/\..*$/, "", T[6]) sub (/_[^_]*$, "", T[6]) print T[6] }
awk: cmd. line:1:                                                                                                                      ^ unexpected newline or end of string

worked output: maybe the awk has to modified accordingly

You can't stuff all statements from a structured, indented script into a single line just one after the other and expect it to work; at least semicolons have to be added to separate the statements, and a logics check should be done afterwards as well.

But, you are right, a / was missing in the command. I updated the post accordingly, thanks.

Besides that, not having any data to work upon, above proposal was an approximation to be tested and adapted - it was a starting point to show the concept.
BTW, your specification in post#1 did not match what was coded in your script. Starting with a bad specification will never end with good results.

Sorry for the confusion caused.

have replaced code of Post #1

updatefile=$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 |  awk 'NF{NF-=1}1' FS='_' OFS='_' ); yupd=`echo $updatefile"_"$ydate`; 
completiontime=( $(grep $yupd /home/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) ); 

with

updatefile=$(awk -v var=$i  '1; $0 ~ var {exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d ' ' -f3); 
completiontime=( $(grep $updatefile /home/lim/config/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) );

which serve me the purpose of the script.
I am trying to working out the code you have given. This is now my output looks like now: