Extracting log entries from a date onwards

Shell Programming and Scripting
1

One of the log file looks like entries as below.

Wed Apr  6 14:51:18 2011 [pid 24366] [wasadmin] FAIL LOGIN: Client "9.191.21.54"
Wed Apr  6 14:52:53 2011 [pid 25554] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:52:54 2011 [pid 25553] [wasadmin] OK LOGIN: Client "9.191.21.54"
Wed Apr  6 14:55:10 2011 [pid 27442] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:55:12 2011 [pid 27441] [wasadmin] FAIL LOGIN: Client "9.191.21.54"
Wed Apr  6 14:56:12 2011 [pid 28291] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:56:13 2011 [pid 28290] [wasadmin] OK LOGIN: Client "9.191.21.54"
Wed Apr  6 15:11:57 2011 [pid 8774] CONNECT: Client "9.191.21.54"
Wed Apr  6 15:11:57 2011 [pid 8773] [wasadmin] OK LOGIN: Client "9.191.21.54"
Thu Apr  7 19:02:54 2011 [pid 18139] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:00 2011 [pid 18138] [wasadmin] FAIL LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:09 2011 [pid 18146] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:15 2011 [pid 18145] [rijesh] OK LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:21 2011 [pid 18484] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:24 2011 [pid 18483] [wasadmin] OK LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:29 2011 [pid 18492] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:35 2011 [pid 18491] [wasadmin] FAIL LOGIN: Client "9.122.71.33"

From the above file, I want to develop a shell script that extract all lines lines from the above log file from the position onwards last time it executed(The last execution time will be recorded). For instance the log file may have thousands of line; the intention is to run the shell script as cronjob with a frequency of 15 minutes. So when the first time cronjob runs it will read all the files and the subsequent execution it needs to extract the lines of log entries from and after the time of script execution.

Having said the above requirement; I am looking for one piece of information here in this forum. I can get the execution time of the script in time format lets say using "date '+%T'". For example if I am executing script (cronjob) in the time of 19:00:00; then extract all the lines using grep that has entries after logged in 19:00:00 .I can have a logic of reading the log file line by line by incorporating a logic of incrementing execution time. However I suspect that may make the operation expensive if the log file has too many entries. So I am looking for a logic that is competitively less expensive operation. Any advice in this regards would be great help.

Thanks in advance,
Rijesh.

2

Could you post what you have done so far?

3

You could try something like this to get the time stamps into a Unix format. Then compare that to the stored value.

while read LINE; do
  DATESTR=$(echo $LINE | cut -b -23)
  TIME=$(date -d "$DATESTR" +%s)
  echo $TIME
done

It could get time consuming on a large file though...