awk logic for tailing and printing

if the last line of an output contains a certain string 'FAILED', i want to print 200 lines from the output.

here's where I got stuck:

process blah blah blah | awk '{if ($0 ~ /FAILED/) y=x "\n" $0; x=$0};END{print y}'

the above only prints the last 2 lines, and it also searches the entire output for the word "FAILED". i dont want to search the entire output as it can be pretty large. i just want to check the last line of the output, and if this last line contains the word "FAILED", i want the awk to then print the last 200 lines of the output.

so basically, printing the last 200 lines of the output should be dependent only on IF the last line of the output contains the word "FAILED".

without input its really difficult for us to guess what you are trying to do

You can do something like this

Not Tested just wild guess

$ process blah blah blah |awk '/Failed/{end=NR;start=NR-200;next}FNR>=start && FNR<=end{print}' file file

suppose i'm tailing the /var/log/syslog file, i tried the above and it doesn't appear to work.

tail -500 /var/log/syslog | awk '/FAILED/{end=NR;start=NR-200;next}FNR>=start && FNR<=end{print}'

i left out the the "file file" in your example because that wont be necessary. what I want to do is output the 200 lines if and only if the last line of the file contains an entry and the entry contain "FAILED". some times the last line of a file or output can be blank. so i also need to account for that.

btw, the "tail -500" in my example is just a good example that explains the type of manner if which the data i'm working on is given to me. so basically, i would have to run the awk command on the output produced by a process.

Please provide input. you can use attachment option.

the output really doesn't matter as it is in no way the same every time. it varies. that's why i gave the tailing of the syslog as an example. you never know what the format of the output will be. and it really doesn't matter as we're only concerned about whether or not the output contain a string.

Try:

awk '{n=$0; p=p n ORS} NR>m{ sub(/[^\n]*\n/,x,p)} END{if(n~s) printf "%s",p}' m=200 s=FAILED 
1 Like

Still on guess

Try:

Just Test

$ echo | awk 'BEGIN{for(i=1;i<=300;i++)print i}END{print "Failed"}' | \
awk '{print >"tmp"}END{if(/Failed/)print NR }' | \
while read line; do [ ! -z "$line" ] && awk -v start=$line 'NR>=(start-200) && NR<start' tmp && rm -rf tmp;done

In your case

$ Process blah blah | awk '{print >"tmp"}END{if(/Failed/)print NR }' | \
while read line; do [ ! -z "$line" ] && awk -v start=$line 'NR>=(start-200) && NR<start' tmp && rm -rf tmp;done

this worked!!!

anyway I can add a "else" into it to print the entire output in the event the last line of the output doesn't contain the "FAILED" string?

so the logic i'm after is,

if the last line of the output contains a string, print x amount of lines in the output received. this requirement has been satisfied with your awk code.

the next logic is, if the last line of the log does not contain the string "FAILED", then print the entire output. no need to print a select number of lines. just print everything

Try this might help

$ process Blah Blah | awk '{print >"tmp"}END{print /Failed/ ? NR : 99 }' | \
while read line; do [ "$line" -ne 99 ] && awk -v start=$line 'NR>=(start-200) && NR<start' tmp && rm -rf tmp || cat tmp && rm -rf tmp;done

OR

$ process Blah Blah | awk '{print >"tmp"}END{print /Failed/ ? NR : 99 }' | \
while read line; do [ "$line" -ne 99 ] && tail -200 tmp && rm -rf tmp || cat tmp && rm -rf tmp;done
1 Like

You could try:

awk '{n=$0 ORS; p=p n; f=f n} NR>m{ sub(/[^\n]*\n/,x,p)} END{printf "%s", n~s?p:f}' m=200 s=FAILED

But if the output is really lengthy it might run out of memory.

------edit------
A bit more efficient:

awk '{A[NR]=$0} END{b=A[NR]~s?NR-m+1:1; for(i=(b>0)?b:1; i<=NR; i++) print A}' m=200 s=FAILED
1 Like

@ Scrutinizer
whether this ?

$ awk '{print >"tmp"}END{ system(sprintf("%s", /FAILED/? "tail -200 tmp" : "cat tmp" ))  }'

to delete tmp

$ awk '{print >"tmp"}END{ system(sprintf("%s", /FAILED/? "tail -200 tmp && rm -rf tmp" : "cat tmp && rm -rf tmp" ))  }'

@Akshay, yes something like that should work. I would avoid using rm -rf like that (risky), simply rm . Also you need to know if the temporary file exists and if not use something else, or use a temporary file name that is guaranteed not to exist, and typically use something in the /var/tmp directory..

this is just perfect!

i'm curious, how did you become so proficient at awk? did you take a specific class? i'm pretty sure they dont go into any significant lenght of training on it at any colleges. so i wanna know how you became so good at it that you can pretty much come up with solutions to any problem (using awk). is awk your favorite programming language? if not, what is? i know this has nothing to do with this thread, i'm just curious.

@Scrutinizer
Yes as you said rm -rf is risky and even its better to use something else instead of tmp . I think from this (post #11 system call) we can overcome from out of memory problem, if file is very lengthy right ?

Try this; not sure if this will be efficient on large files...

tac file | awk 'NR==1 && !/FAILED/ {exit} NR>200 {exit} 1' | tac
1 Like

@ RudiC brilliant ! nice solution

But if string FAILED is not found then it does not print anything

for example :

This prints 200 lines

$ awk 'BEGIN{for(i=1;i<=300;i++)print i;print "FAILED"}' | tac | awk 'NR==1 && !/FAILED/ {exit} NR>200 {exit} 1' | tac

whereas this does not

$ awk 'BEGIN{for(i=1;i<=300;i++)print i;print "FAILED"}' | tac | awk 'NR==1 && !/FFAILED/ {exit} NR>200 {exit} 1' | tac

Modified version of RudyC's solution

tac file | awk 'NR==1  &&  !/FAILED/{s=1} s || NR < 200' | tac 

Tested like this
This prints 200 lines

$  awk 'BEGIN{for(i=1;i<=300;i++)print i;print "FAILED"}' | tac | awk 'NR==1 && !/FAILED/{s=1} s || NR <= 200' | tac 

Whereas this prints all

$ awk 'BEGIN{for(i=1;i<=300;i++)print i;print "FAILED"}' | tac | awk 'NR==1 && !/FFAILED/{s=1} s || NR <= 200' | tac 

It should not print anything if FAILED is not found.

@RudiC as you know here requirement is keep on changing...:slight_smile:

from #8

Rats! Overlooked that - sorry, need to rethink...

EDIT: Well, try that:

tac file1 | awk 'NR==1 {if(/FAILED/) L=200; else L=1E23}  NR<=L' | tac

or even

tac file1 | awk 'NR==1 {L=(/FAILED/)?200:1E23}  NR<=L' | tac

Actually your tac approach is really nice on large file compare my system call inside awk ...

I just modified your approach like this

tac file | awk 'NR==1  &&  !/FAILED/{s=1} s || NR < 200' | tac