Extract data from log file in specified range of time

Shell Programming and Scripting
1

I was searching for parsing a log file and found what I need in this link

But the most useful answer (posted by @Kent):

# this variable you could customize, important is convert to seconds. 
# e.g 5days=$((5*24*3600))
x=$((5*60))   #here we take 5 mins as example

# this line get the timestamp in seconds of last line of your logfile
last=$(tail -n1 logFile|awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' )

#this awk will give you lines you needs:
awk -F'[][]' -v last=$last -v x=$x '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; if (last-d<=x)print $0 }' logFile

is giving the following error:

[centos@localhost parse log file]$ bash parse.sh
sh: -c: line 0: unexpected EOF while looking for matching `"'
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `"'
sh: -c: line 1: syntax error: unexpected end of file
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:20:41 +0200  "GET 
### lines below are what you want (5 mins till the last record)
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:27:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:30:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:30:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:30:41 +0200  "GET 
172.16.0.3 - -  31 Mar 2002 19:30:41 +0200  "GET

this is the logFile that is being parsed using this bash script

172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:20:41 +0200] "GET 
### lines below are what you want (5 mins till the last record)
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:27:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:30:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:30:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:30:41 +0200] "GET 
172.16.0.3 - - [31/Mar/2002:19:30:41 +0200] "GET

I spend lot of time trying to solve before I ask here but didn't find any solution.

The script will be called by the crontab to get the last 1 min log lines and count how many times an ip is listed in one minute so I can detect if it is an attack or not. and this is another task hope that an expert will help giving the needed code here in the same question.(I think it can be solved in 2 lines but I most search the web to find how to count distinct string in a textfile).

and here is ana image showing the code the logfile and the result.

2

Please check this code


# this variable you could customize, important is convert to seconds. 
# e.g 5days=$((5*24*3600))
x=$((5*60))   #here we take 5 mins as example

# this line get the timestamp in seconds of last line of your logfile
last=$(tail -n1 logFile|awk -F'[][]' '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; print d;}' )

#this awk will give you lines you needs:
awk -F'[][]' -v last=$last -v x=$x '{ gsub(/\//," ",$2); sub(/:/," ",$2); "date +%s -d \""$2"\""|getline d; if (last-d<=x)print $0 }' logFile
3

no differences between the code you post and mine