abhii
October 20, 2016, 3:02am
1
Hi All,
I am trying to perform certain task using awk command but the command is not returning expected output though it is running fine independently but not in for loop.
req:
I am searching a pattern (say pattern1) in a dated file
trying to search the last occurrence of the another pattern (say pattern2) before last occurrence of pattern1.
what i am trying do:
search the file name where pattern1 is pattern (for today's date).
print the file from start to the last occurrence of pattern1
search first occurrence of pattern2 from the last occurrence of pattern1.
here is code snippet:
for i in PUAFZ00 AATGY00 AATHB00 AATGZ00;
do
echo =======================================================================================================================================;
echo -e " Symbol - $(tput setaf 1) $i ";
for d in Mon Tue Wed Thu Fri;
do
ydate=`date +%Y%m%d -d "last $d"`;
filename=`grep $i /updates/logs/log*$ydate* | cut -d ':' -f1 | uniq`;
updatefile=$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 | awk 'NF{NF-=1}1' FS='_' OFS='_' ); yupd=`echo $updatefile"_"$ydate`;
completiontime=( $(grep $yupd /home/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) );
echo "$(tput sgr 0) Processing_date - $(tput setaf 2) $ydate $(tput sgr 0) UpdatePackage - $(tput setaf 6) $updatefile $(tput sgr 0) Processing_Time $(tput setaf 3)${completiontime[@]:0} $(tput sgr 0)" ;
done;
echo =======================================================================================================================================; done
Unexpected output received on command:
$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 | awk 'NF{NF-=1}1' FS='_' OFS='_' );
If i pass this command by passing the values on prompt expected output is returned.
Without seeing any sample data, seeing the results that are being produced, and seeing the results that are desired (all in CODE tags); it is very difficult to make any suggestions about what code changes should be made.
RudiC
October 20, 2016, 3:48am
3
On first sight it pops up that within single quotes shell variables will not be expanded, so
$(awk '1;/$i/{exit}' $filename
will not work as expected.
abhii
October 20, 2016, 4:43am
4
for i in PUAFZ00;
do
echo =======================================================================================================================================;
echo -e " Symbol - $(tput setaf 1) $i ";
for d in Mon;
do
ydate=`date +%Y%m%d -d "last $d"`;
filename=`grep $i /home/lim/updates/logs/log*$ydate* | cut -d ':' -f1 | uniq`;
updatefile=$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 | awk 'NF{NF-=1}1' FS='_' OFS='_' ); yupd=`echo $updatefile"_"$ydate`;
completiontime=( $(grep $yupd /home/lim/config/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) );
echo "$(tput sgr 0) Processing_date - $(tput setaf 2) $ydate $(tput sgr 0) UpdatePackage - $(tput setaf 6) $updatefile $(tput sgr 0) Processing_Time $(tput setaf 3)${completiontime[@]:0} $(tput sgr 0)" ;
done;
echo =======================================================================================================================================; done
output:
Symbol - PUAFZ00
Processing_date - 20161017 UpdatePackage - upd_0_k36 Processing_Time 21:25:21 EDT 2016
When i run the awk command indepndently; i get the result: (this is expected)
grep command
grep PUAFZ00 /home/lim/updates/logs/log*`date +%Y%m%d -d "last Mon"`* | cut -d ':' -f1 | uniq
output:
/home/lim/updates/logs/log.0_20161017.old
awk command:
awk '1;/PUAFZ00/{exit}' /home/lim/updates/logs/log.0_20161017.old | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 | awk 'NF{NF-=1}1' FS='_' OFS='_'
output awk:[this is expected one]
Unpack complete upd_0_pluz
---------- Post updated at 03:09 AM ---------- Previous update was at 02:56 AM ----------
awk -v var="$i" '1;/var/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 | awk 'NF{NF-=1}1' FS='_' OFS='_'
is also returning the same output:
UpdatePackage - upd_0_k36
---------- Post updated at 03:43 AM ---------- Previous update was at 03:09 AM ----------
awk -v var=$i -v var2="$filename" '1;/var/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}'
output:
RudiC
October 20, 2016, 4:45am
5
This is one step further, but still not enough. /var/
will look for the string "var" verbatim.
Try
$0 ~ var
.
abhii
October 20, 2016, 4:47am
6
rudic:
On first sight it pops up that within single quotes shell variables will not be expanded, so
$(awk '1;/$i/{exit}' $filename
will not work as expected.
awk -v var=$i -v var2="$filename" '1;/var/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 | awk 'NF{NF-=1}1' FS='_' OFS='_'
awk -v var=$i -v var2="$filename" '1;/var/{exit}' var2 | awk '/upd_/ {a=$0} END{print a}'
Please help on this:
awk -v var=$i -v var2="$filename" '1;/$0/{exit}' var2 | awk '/upd_/ {a=$0} END{print a}'
RudiC
October 20, 2016, 5:03am
7
Try
awk -v var=$i '1; $0 ~ var {exit}' $filename
You need to finely differentiate between what is IN the awk
command, and what is outside (i.e. shell).
1 Like
RudiC
October 20, 2016, 5:27am
9
Looking at it again, the looong pipe could be reduced to one single awk
command, although without decent input sample this is impossible to verify. Try
awk -v var=$i '
/upd_/ {a = $0} # keep last line containing "upd_" before the one -
$0 ~ var {exit} # containing the pattern in "var"; when found, leave
# script jumping to END section
END {split (a, T, "/") # spread "a" variable into "T" array on "/"
sub (/\..*$/, "", T[6]) # remove everything after first dot "."
sub (/_[^_]*$/, "", T[6]) # remove the last part after last "_" in element
print T[6] # print result in element
}
' $filename
and come back with the results.
EDIT: /
added after abhii's comments/errors...
abhii
October 20, 2016, 6:50am
10
rudic:
Looking at it again, the looong pipe could be reduced to one single awk
command, although without decent input sample this is impossible to verify. Try
awk -v var=$i '
/upd_/ {a = $0} # keep last line containing "upd_" before the one -
$0 ~ var {exit} # containing the pattern in "var"; when found, leave
# script jumping to END section
END {split (a, T, "/") # spread "a" variable into "T" array on "/"
sub (/\..*$/, "", T[6]) # remove everything after first dot "."
sub (/_[^_]*$, "", T[6]) # remove the last part after last "_" in element
print T[6] # print result in element
}
' $filename
and come back with the results.
for me
awk -v var=$i '1; $0 ~ var {exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d ' ' -f3
did work to get the filename with date format (what was required)
though the code you have posted given me the error:
awk -v var=$i '/upd_/ {a = $0} $0 ~ var {exit} END {split (a, T, "/") sub (/\..*$/, "", T[6]) sub (/_[^_]*$, "", T[6]) print T[6] }' $filename
awk: /upd_/ {a = $0} $0 ~ var {exit} END {split (a, T, "/") sub (/\..*$/, "", T[6]) sub (/_[^_]*$, "", T[6]) print T[6] }
awk: ^ unterminated regexp
awk: cmd. line:1: /upd_/ {a = $0} $0 ~ var {exit} END {split (a, T, "/") sub (/\..*$/, "", T[6]) sub (/_[^_]*$, "", T[6]) print T[6] }
awk: cmd. line:1: ^ unexpected newline or end of string
worked output: maybe the awk has to modified accordingly
RudiC
October 20, 2016, 7:14am
11
You can't stuff all statements from a structured, indented script into a single line just one after the other and expect it to work; at least semicolons have to be added to separate the statements, and a logics check should be done afterwards as well.
But, you are right, a /
was missing in the command. I updated the post accordingly, thanks.
Besides that, not having any data to work upon, above proposal was an approximation to be tested and adapted - it was a starting point to show the concept.
BTW, your specification in post#1 did not match what was coded in your script. Starting with a bad specification will never end with good results.
abhii
October 20, 2016, 7:39am
12
Sorry for the confusion caused.
have replaced code of Post #1
updatefile=$(awk '1;/$i/{exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d '/' -f6 | cut -d '.' -f1 | awk 'NF{NF-=1}1' FS='_' OFS='_' ); yupd=`echo $updatefile"_"$ydate`;
completiontime=( $(grep $yupd /home/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) );
with
updatefile=$(awk -v var=$i '1; $0 ~ var {exit}' $filename | awk '/upd_/ {a=$0} END{print a}' | cut -d ' ' -f3);
completiontime=( $(grep $updatefile /home/lim/config/load_updates.hst | grep "Unpack complete" | cut -d " " -f5-7) );
which serve me the purpose of the script.
I am trying to working out the code you have given. This is now my output looks like now:
=======================================================================================================================================
Symbol - PUAFZ00
Processing_date - 20161017 UpdatePackage - upd_0_pluz_20161017 Processing_Time 17:00:48 EDT 2016
Processing_date - 20161018 UpdatePackage - upd_0_pluz_20161018 Processing_Time 21:20:36 EDT 2016
Processing_date - 20161019 UpdatePackage - upd_0_pluz_20161019 Processing_Time 21:45:32 EDT 2016
Processing_date - 20161013 UpdatePackage - upd_0_pluz_20161013 Processing_Time 16:50:37 EDT 2016
Processing_date - 20161014 UpdatePackage - upd_0_pluz_20161014 Processing_Time 16:41:11 EDT 2016