Check specific content from log file

Hi all,

i have a logfile which is continuously being updated. I have built a script to check for a specific content and if it is found, it sends a string into a file. Here's the current script:

#!/bin/bash

logfile=/opt/jboss-eap-6.3/standalone/log/server.log

tail -fn0 $logfile | \
while read line ; do
        echo "$line" | grep -q "SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
        if [ $? = 0 ]
        then
            echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.txt
        else
            echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
        fi
done

As you can see i am continuously reading from the "server.log" file and while reading each entry, if the "grep" command find the matched string it echoes to file, else it echos to the same file but with a different string (YES or NO).

However i noticed that when the end of the day is reached, i.e @ 23:59, the file "/home/nms/Disconnection_oldstack.txt" stops updating. I noticed that the "server.log" file closes and a new one is reopened at that time, here below:

-rw-r--r--. 1 jboss jboss  29304874 Feb 10 23:59 server.log.2017-02-10.gz
-rw-r--r--. 1 jboss jboss  21277719 Feb 11 23:59 server.log.2017-02-11.gz
-rw-r--r--. 1 jboss jboss 445767175 Feb 12 23:59 server.log.2017-02-12
-rw-r--r--. 1 jboss jboss 782710795 Feb 13 23:59 server.log.2017-02-13
-rw-r--r--. 1 jboss jboss 143127996 Feb 14 09:40 server.log

i tried to add

while [ ! -f $logfile ]; do sleep 5; done

in the beginning of the script so it sleeps if file is does not exist (since i thought that at the time of the file rotation it stops since the file is not found), but the same thing happened.
Am i missing something?
Note: The output redirection to file is necessary in this case so i need to stick to that!

Thanks in advance for your feedback!

Which OS and version are you using? The output from uname -a would be very helpful.

You might find that tail has a -F flag in your version. have a look in the man page and see if it is supported.

I hope that this helps,
Robin

1 Like

If I understand what is happening, the process that builds the server.log file closes the file at 23:59. It then renames the file and open a new server.log file.

I know you tried to wait a few seconds; what happens if you try to wait longer?

While the old file may close and rename, perhaps the new file does not get created until the first event to cause something to be written to it.

Unsure on what part of that sript makes it continuously read. The code snippet seems to be a one-time pass-thru.

1 Like

tail -f

1 Like

Elaborating on what rbatte1 already alluded to above: the man page for tail (GNU coreutils) 8.25 reads:

Which I guess would exactly solve your problem.

3 Likes

Hi,

Thanks for your feedback. I modified the script to use -F instead and increased the sleep time (although this might not be really needed since -F helps in cases rotated log files.
We'll see how it goes and keep you posted.
fyi the OS is a CentOS release 6.6

#!/bin/bash

logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while [ ! -f $logfile ]; do sleep 60; done

tail -F $logfile | \
while read line ; do
        echo "$line" | grep -q "SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
        if [ $? = 0 ]
        then
            echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.txt
        else
            echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
        fi
done

I'm not sure if you expoect lots of messages to rush through at any time, but that could be painful calling grep for each and every one one.
If the string you are checking is exactly what you show (no leading date string or other stuff) might it be better to code like this:-

#!/bin/bash

alert_string="SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while read line
do
   if [ "$line" = "$alert_string" ]
   then
      echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.
   else
      echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
   fi
done < <(tail -F $logfile)

If the string you are looking for is only part of the line, then this might be better:-

#!/bin/bash

alert_string="SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while read line
do
   test_line="${line%${alert_string}*}"
   if [ "$line" != "$test_line" ]
   then
      echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.
   else
      echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
   fi
done < <(tail -F $logfile)

Both of these negate the need to make the external call to grep The second one uses variable substitution to cut off the string from the end of the line so if there is a leading timestamp, that will be put into variable test_line and the comparison done , which will then go to the then section. if the alert string is not in the line, then the whole line is put into test_line and the comparison will match so we go to the else section.

I did also wonder if you really want to be overwriting the output file each time your read a record too, as this could cause a heavy IO load to re-write a single line in the file many times over. If you keep the current state in a variable, you can also avoid the unnecessary IO and only re-write the file each time the state changes.

Perhaps this would do both:-

#!/bin/bash

alert_string="SocketTimeoutException invoking http://10.192.1.8:8001: connect timed out"
alert_state=""
logfile=/opt/jboss-eap-6.3/standalone/log/server.log

while read line
do
   test_line="${line%${alert_string}*}"
   if [ "$line" != "$test_line" ]
   then
      if [ "$alert_state" != "YES" ]
      then
         echo "ConnectionTimeout_ESB-OldStack=YES" > /home/nms/Disconnection_oldstack.
         alert_state="YES"
   else
      if [ "$alert_state" != "NO" ]
      then
         echo "ConnectionTimeout_ESB-OldStack=NO" > /home/nms/Disconnection_oldstack.txt
         alert_state="NO"
   fi
done < <(tail -F $logfile)

If you just want to append to the output file, put the output redirection on the whole loop, i.e. after the done and the input redirection.

I hope that this helps,
Robin

1 Like

Hi Robin, many thanks for your input. The script now works using the tail -F option. In fact there was no need to perform the while loop to sleep.

Caught my attention in the heavy IO. Makes sense not to overwrite and just input when the state changes.

Thanks all for your help!