Script to count matched string then check again from last read position

Shell Programming and Scripting
1

Hi all,

I am currently trying to figure out how can i capture a particular string from a log file, then search again but from the last line it read before onward.
Basically let's say that the check runs every 5 mins via cron, counting the number of matched strings "Cannot assign requested address" from the log sample below:

Nov 28 18:18:09.005100 diameterControlAgent(3426) NOTICE: {960112} Peer: 'jinny': adding permitted origin host 'AM1RTC01.NL.ASPIDERSOLUTIONS.COM'
Nov 28 18:18:09.005649 diameterControlAgent(3426) NOTICE: {960112} Peer: 'jinny': adding permitted origin host 'AM1RTC02.NL.ASPIDERSOLUTIONS.COM'
Nov 28 18:18:09.009531 diameterControlAgent(3426) NOTICE: {960113} Peer: 'jinny': adding Remote Address '<ip address>'
Nov 28 18:18:09.010373 diameterControlAgent(3426) NOTICE: {960113} Peer: 'jinny': adding Remote Address 'ip address>'
Nov 28 18:18:09.010873 diameterControlAgent(3426) NOTICE: {960114} Loading peer: 'jinny': netmaskBits: 32
Nov 28 18:18:09.011362 diameterControlAgent(3426) NOTICE: {960115} Loading peer: 'jinny': permittedInstances: 0
Nov 28 18:18:09.011981 diameterControlAgent(3426) NOTICE: {960116} Loading peer: 'jinny': reqSctpInboundStreams: 8
Nov 28 18:18:09.012590 diameterControlAgent(3426) NOTICE: {960117} Loading peer: 'jinny': reqSctpOutboundStreams: 8
Nov 28 18:18:09.014542 diameterControlAgent(3426) NOTICE: {960013} Configuration Loaded
Nov 28 18:18:09.016790 diameterControlAgent(3426) ERROR: {969108} tcp:::<ip address:port>: bind to <ip address:port> failed: Cannot assign requested address.

The next time the script runs again, the aim is to continue searching for this string from the last "line" (i.e end of file) it read previously.

Hence either using AWK or grep -c. The problem s is how to make the script intelligent enough to search from the last read line onward. The log file changes continuously.

is it possible to achieve this?

I was thinking maybe using either the tail command to store the output then match it, but am afraid that most probably there would be similar matches.
Else maybe try a word count?

Do you have ideas?

2

If you know the last read line number, you can use sed to get the lines past that point:-

last_seen_record=12345            # .... or whatever you have determined it to be
sed -n "$last_seen_record,$"p  $logfile

Of course, the output is then from the point you specify onward, so working out the next last_see_record value will be tricky. You might be better with:-

last_seen_record=12345            # .... or whatever you have determined it to be
grep -c "" $logfile | sed -n "$last_seen_record,$"p

This will give you output prefixed with the line number of the whole file so you can work with that to save away the appropriate value for next time.

If you have large log files, you might hit a timing problem if you try to count the number of lines read as a separate operation with wc because of the delay between counting the lines and the process that you your to read the information out.

An alternate would be to use awk to both extract the data and keep a record count in a single operation and we can explore that if you think that is more appropriate, but it can be a little harder to grasp.

Can you post the output from uname -a so we know what OS and version we're working with, because there are variations we may need to consider.

Robin

3

Hi Robin,

As for knowing the last read number, i guess i can ontain that by using the current number of lines using wc or a one liner with perl

perl -lne 'END { print $. }' $LOG_FILE

but the problem i'm trying to figure out is how to let the script know how to read from the last read line onward

#File path
LOG_FILE=/IN/service_packages/DCA/tmp/diameterControlAgent_rtcg_tmnl.log
#Current lines
last_record=`perl -lne 'END { print $. }' $LOG_FILE`
#Get count
grep -c "" $LOG_FILE | sed -n "$last_record,$"p

Of course the above is not correct. The grep will still count from the beginning of the file.

The OS here is Solaris 11. SunOS 5.11 11.2 sun4v sparc sun4v

4

Not sure I understood correctly what you're after, but if you want to continue processing a file from where you stopped last time, this post and the reference therein might be interesting...

5

An alternate may be to open the file with tail and keep it open.

What is your overall aim for this process? You might be able to:-

while read line
do
   process "$line"
done < <(tail -f $logfile)

I think that the < <(command) is a bash construct rather than sh or ksh so it depends if you have that or even if this is appropriate. This process will not end, but continue reading the log file when new records are written, so it's near-time (strictly not quite real-time)

Beware that the log file will remain open, so if you rotate your log files, you might need to use tail -F $logfile instead to open the replacement file.

Can you tell us what the point of this code is to be? There may be a simpler way that I'm not thinking of. It would be useful to know the OS version (output from uname -a ) and shell that you are using.

Kind regards,
Robin

6

Not sure exactly what you want to do with the output (maybe send an email ??) but the below script will search for the text and then if found send an email and then continue on from the last line searched.

#!/bin/ksh

while true
do
line_no=$(wc -l log_file | awk '{line=$1 +1; print line}')
sleep 20
tail +$line_no log_file| grep "Cannot assign requested address" > /tmp/err.txt
    if [ $? -eq 0 ]
then mailx -s "There is an error" emailaddress < /tmp/err.txt
    fi
done