Script to grep if 404 error got generated in the log files in last 5 minutes

UNIX for Beginners Q & A
1

I have a log file which are linked with prefix current date and suffix with IP_address.log 2020_04_22.shared_http_server.192.168.XX.XX.log

I want to monitor these logs at an interval of every 5 minutes. I am looking to schedule a script at an interval of every 5 minutes,in order to grep if 404 error got generated in the log files in last 5 minutes and send the lines of the logs which contains 404 error in different file.

I have written a script to get this done and i am using cron to run this script for every 5 min.Whenever the script runs, it reads the whole log file and throws all the lines of the logs which contains 404 error in different file.But, i want those lines of the logs containing 404 error message to be sent in different file which got generated in last 5 minutes.

find /home/pranav -type f -name "2020_04_22.shared_http_server.192.168.XX.XX.log" |while read file
  do
    RESULT=$(egrep "[^0](404)" $file)
      if [[ ! -z $RESULT ]]
         then
            echo "Error(s) in $file on $HOSTNAME at "$(date)": $RESULT">> log_result.txt
     fi
  done

This script is able to grep the lines which contains 404 error message but i want those lines of the logs containing 404 error message to be sent in different file which got generated in last 5 minutes.

Could somebody help me on this what i'm i missing here?
Thank you in advance.

2

Could you store a 'previous last line' value somewhere to use between runs? That way you could:-

  • Copy the log file to a temporary directory (to get a fixed file to work with)
  • Read the 'previous last line' value from your stored location
  • Count the lines in your copied file.
  • If the count is lower, you have rotated the log file, so set the 'previous last line' value to zero to read the whole file.
  • If the count was larger, read from the 'previous last line' value to the end of the file looking for the error message being logged.
  • Store the new 'last line' value in a file somewhere for the next run.
  • Tidy up your temporary file/directory.

Would that logic work for you?

Let us know if that helps, is totally wrong or if you get stuck.

I hope that this helps,
Robin

2 Likes