To check timestamp in logfile and display lines upto 3 hours before current timestamp

Hi Friends,

I have the following logfile. Currently time in india is 07/31/2014 12:33:34 and i have the following content in logfile. I want to display only those entries which contain string 'Exception' within last 3 hours. In this case, it would be the last line only

[7/30/14 4:59:30 Exception Found
[7/30/14 5:18:55 Result is OK
[7/30/14 8:45:22 Exception found
[7/30/14 9:22:26 Exception found
[7/31/14 11:55:66 Exception found

I can get the timestamp in logfile as

awk '/Exception/ {print $1,$2}' trial.txt | sed 's/^.//' which gives below results

7/30/14 4:59:30
7/30/14 8:45:22
7/30/14 9:22:26
7/31/14 11:55:66

And i can get current timestamp as

date +"%m/%d/%Y %T"
07/31/2014 12:33:34

So, how can i compare current timestamp with timestamp in log file and display 'Exception' entries until last 3 hours.

---------- Post updated at 02:40 AM ---------- Previous update was at 02:07 AM ----------

I am trying this command. But it is throwing exception saying syntax error. What am i doing wrong?

awk '/Exception/ if ($1,$2 > $(date +"%m/%d/%Y %T" -d  "3 hour ago")) print $1,$2 ' trial.txt  | sed 's/^.//'

There are several faults:

• date +%Y gives YYYY, but your logfile has YY
• $1,$2 won't work. $1" "$2 is a concatenation of 3 strings
• $(date...) only works inside " " but not ' '
• A string comparison only works if both the strings have leading zeros
• A string comparison needs the order "YY MM DD"

Here is an awk solution that reformats with leading zeros and correct order,
and date +%y gives the short year (not Y3k compliant).

etime=$(date +"%m/%d/%y %T" -d  "3 hour ago")
awk -v etime="$etime" '
BEGIN {
  split(etime,Z)
  split(Z[1],ZD,"/")
  split(Z[2],ZT,":")
  etime=sprintf("%02d/%02d/%02d %02d:%02d:%02d",ZD[3],ZD[2],ZD[1],ZT[1],ZT[2],ZT[3])
}
/Exception/ {
  sub(/^\[/,"")
  split($1,D,"/")
  split($2,T,":")
  time=sprintf("%02d/%02d/%02d %02d:%02d:%02d",D[3],D[2],D[1],T[1],T[2],T[3])
  if (time"" > etime"") {exit}
  print $1,$2
}
' trial.txt

Hey thanks germany, but the output is showing all the lines with exception. Right now, it should not show any lines.

$ awk -v etime="$etime" '
> BEGIN {
>   split(etime,Z)
>   split(Z[1],ZD,"/")
>   split(Z[2],ZT,":")
>   etime=sprintf("%02d/%02d/%02d %02d:%02d:%02d",ZD[3],ZD[2],ZD[1],ZT[1],ZT[2],ZT[3])
> }
> /Exception/ {
>   sub(/^\[/,"")
>   split($1,D,"/")
>   split($2,T,":")
>   time=sprintf("%02d/%02d/%02d %02d:%02d:%02d",D[3],D[2],D[1],T[1],T[2],T[3])
>   if (time"" > etime"") {exit}
>   print $1,$2
> }
> ' trial.txt

7/30/14 4:59:30
7/30/14 8:45:22
7/30/14 9:22:26
7/31/14 11:55:66

Well, try this based on MadeInGermany's proposal:

 awk -vDT=$(date +"%y%m%d%H%M" -d"- 3 hour") '
                 {sub(/^\[/,"")
                  split ($1, D, "/")
                  split ($2, T, ":")
                  AT=sprintf ("%02d%02d%02d%02d%02d", D[3], D[1], D[2], T[1], T[2])}
         AT > DT && /Exception/
        ' file
7/31/14 11:55:66 Exception found

which is fine as my local time is 31.07.14 14:07.

If I'm reading MadeInGermany's code correctly, it is comparing YY/DD/MM HH:MM:SS instead of YY/MM/DD HH:MM:SS and is printing the more than 3 hours ago timestamps instead of the less than 3 hours old timestamps. As long as we're constructing strings to compare, I don't see the need to include the slashes in the dates and I'm also assuming that the minutes and seconds do have 2 digits with zero fill so I don't have to split the time fields (I just have to use leading 0 to fill an 8 character field to supply missing leading zeroes in the hour). And, I used FS instead of sub() and split() to split the date field.

I think RudiC left out a %S in the date format string, but on a 3 hour window, a difference of up to one minute might not be noticeable in the results.

I think this does what was requested (on systems where the date utility supports this form of -d option processing):

awk -v d="$(date "+%y%m%d%T" -d "3 hours ago")" -F '[[/ ]' '
/Exception/ {
        if(sprintf("%02d%02d%02d%08s", $4, $2, $3, $5) > d)
		printf("%s/%s/%s %s\n", $2, $3, $4, $5)
}' trial.txt

This is perfect, but im not able to store the result by logging into another server. What am i doing wrong?

 
VAR1=$(ssh server1 "awk -vDT=$(date +"%y%m%d%H%M" -d"- 3 hour") '
                 {sub(/^\[/,"")
                  split ($1, D, "/")
                  split ($2, T, ":")
                  AT=sprintf ("%02d%02d%02d%02d%02d", D[3], D[1], D[2], T[1], T[2])}
         AT > DT && /Exception/
        ' file")

Exception:

 
awk: cmd. line:1:                  {sub(/^\[/,)
awk: cmd. line:1:                             ^ syntax error
awk: cmd. line:1: fatal: 0 is invalid as number of arguments for sub

Passing complex commands with ssh is problematic, because there are two shells that evaluate the script: one on the local host and one on the remote host.
Save the script on the local host, and pass it via stdin to the remote shell:

ssh -x server1 "/bin/sh -s" < savedscript

The -s option allows to place script arguments

ssh -x server1 "/bin/sh -s arg1 arg2" < savedscript

Thanks, its working perfect. Can you tell me how to get the count of the logs . What field do i need to edit in the below code.

 
awk -vDT=$(date +"%y%m%d%H%M" -d"- 3 hour") '
                 {sub(/^\[/,"")
                  split ($1, D, "/")
                  split ($2, T, ":")
                  AT=sprintf ("%02d%02d%02d%02d%02d", D[3], D[1], D[2], T[1], T[2])}
         AT > DT && /Exception/
        ' file

Try (untested)

         AT > DT && /Exception/   {CNT++; print}
         END                      {print CNT}

Hey Guys,

Now i have another logfile where the timestamp in logs are displayed in a different manner as

 
2014.09.16 05:15:36.354.MST Exception Found
2014.09.16 08:18:36.277.MST Result Ok
2014.09.16 17:10:22.233.MST Exception Found

This should output the result for last line only as Time currently in india is 17:51 and it is the only string Matching "Exception Found" in the last 3 hours.

How to modify the script propsed earlier to do this?

---------- Post updated 09-17-14 at 01:14 AM ---------- Previous update was 09-16-14 at 07:22 AM ----------

@Rudic , @Germany : I Need your help guys

Given the help you have already received, why don't you show us what you have tried on your own to solve this slightly different problem?

The UNIX and Linux Forum is not a place to get free programming services every time you decide to change your requirements. We want you to learn from the examples we provide so you can do it yourself the next time.

Thanks ,ill try to script this myself. I need some help understanding the below line.

 
AT=sprintf ("%02d%02d%02d%02d%02d", D[3], D[1], D[2], T[1], T[2])

What exactly is the function of Sprintf function and why is it in the order of D[3],D[1],D[2] instead of D[3],D[2],D[1] . If we are comparing the current timestamp (date +"%y%m%d%H%M" -d"- 3 hour") against the timestamp in logs, after splitting the timestamp in logs, it should be in order of yy/mm/dd as well.

---------- Post updated at 06:39 AM ---------- Previous update was at 06:38 AM ----------

Please ignore the above. I just realized my mistake. It will be D[3],D[1],D[2] only.

---------- Post updated at 07:54 AM ---------- Previous update was at 06:39 AM ----------

I modified the script as below and its working fine for the new File

 awk -vDT=$(date +"%Y%m%d%H%M" -d"- 3 hour") '
                     {
                      split ($1, D, ".")
                      split ($2, T, ":")
                      AT=sprintf ("%02d%02d%02d%02d%02d", D[1], D[2], D[3], T[1], T[2])}
             AT > DT && /Exception Found/
        ' /logs/Trace.log

Congratulations! We are always glad to hear that the people who have come to The UNIX and Linux Forums are learning how to write code that works after getting help with the basics from the volunteers here.

And thank you for posting your working code; you are now one of the volunteers showing others how you solved your problem.