Pick the last one hour lines from log matching this pattern.

Shell Programming and Scripting
1

Hello please help me on this,

pick the last one hour lines from the log, which have the prefix time format like this.

[2/5/12 10:50:12:148 EDT] log message
[2/5/12 10:50:13:324 EDT] log message

i tried to do grep, but that failed.

my code

grep '(date +['%-m/%-d/%y %H:%M:%S:%03N %Z'])' log_file_path

This checking only the current time stamp. How to get the log for last one hour from the current time, for this time format.

2

Does your date function support the --date=STRING option?

e.g.

$ date --date="date -d "2/5/12 10:50:12.148 EDT"
Sun, Feb 05, 2012 10:50:12 AM
1 Like
3

The first problem with your code is you used single quotes: everything within single quotes is treated as is, not evaluated. This will probably work, but only find the entries with the current timestamp, not the last hour:

grep "$(date +['%-m/%-d/%y %H:%M:%S:%03N %Z'])" log_file_path

You can get a timestamp from one hour before if you manipulate the TZ variable, which holds the timezone. For instance, i am on GMT+1. To get the timestamp from one hour before I'd use:

(TZ="GMT+1" ; date)    # current time
(TZ="GMT" ; date)      # one hour ago

Lastly, grep is a poor tool for what you want to achieve, because it filters single lines. What you want to match is a range of lines and for this sed is the tool of choice. Notice how the single quoted parts are ending and restarting again:

sed  -n '/^'"$(TZ=GMT ; date +['%-m/%-d/%y %H:%M:%S:%03N %Z'])"'/,$ p' log_file_path

I hope this helps.

bakunin

1 Like
4

Thanks in advance.
I am using kshell. And the code i displayed above also not working.
please help me in getting the lines for the correct format. Can use grep or awk command.

---------- Post updated at 08:44 PM ---------- Previous update was at 08:36 PM ----------

i have given the logfile path /hostname/log/stdout.log at the end of sed command.
getting error cant read path: No such file or directory

---------- Post updated at 08:48 PM ---------- Previous update was at 08:44 PM ----------

Thanks backunin

That worked but displayed the complete log.
Not just the hour back lines.

Please help me.

5

You could try this, assuming your date command supports --date :

v=$(date --date "-1 hour" +"-vY=%-y -vT=%-m -vD=%-d -vH=%-H -vM=%-M")
awk -F'[[ /:]' $v '
  $4>Y ||
  $4==Y&&$2>T ||
  $4==Y&&$2==T&&$3>D ||
  $4==Y&&$2==T&&$3==D&&$5>H ||
  $4==Y&&$2==T&&$3==D&&$5==H&&$6>M {v=1}
  v' logfile
1 Like
6

Hello chubler.
Thanks for the post/
But getting the following syntax error.

awk: =F[[ /:]
awk:^ syntax error

And will this work for the format i displayed in the log sample?

7

Yes, tested with the format posted, some awk versions require a space between -F and parameter try (notice additional space here):

v=$(date --date "-1 hour" +"-vY=%-y -vT=%-m -vD=%-d -vH=%-H -vM=%-M")
awk -F '[[ /:]' $v '
  $4>Y ||
  $4==Y&&$2>T ||
  $4==Y&&$2==T&&$3>D ||
  $4==Y&&$2==T&&$3==D&&$5>H ||
  $4==Y&&$2==T&&$3==D&&$5==H&&$6>M {v=1}
  v' logfile
1 Like
8

Thanks chubler
that awk worked.
But It displayed everything in the log file.
Not just 1 hour back.

9

Can you paste the first line that is displayed here please.
I suspect your datafile doesn't match the posted format particularly check that the year is two digits and not four, ans make sure no additional spaces are in between the dates, eg " 8" for the month.

10

the initial lines are the start of the log file. It did not do awk. displayed everything in the file

*******Start Display Current Environment *
Host OPerating Syatem is linux, version
Java version

11

OK you have lines that don't match the format at the start of the file.

Try:

v=$(date --date "-1 hour" +"-vY=%-y -vT=%-m -vD=%-d -vH=%-H -vM=%-M")
awk -F '[[ /:]' $v '
  $4+0>Y ||
  $4+0==Y&&$2+0>T ||
  $4+0==Y&&$2+0==T&&$3+0>D ||
  $4+0==Y&&$2+0==T&&$3+0==D&&$5+0>H ||
  $4+0==Y&&$2+0==T&&$3+0==D&&$5+0==H&&$6+0>M {v=1}
  v' infile
12

Thanks Chubler,

This avoided the files that do not have timestamp at the start of the file.
But displayed all the lines with the time stamp and also lines in the missdle without timestamp.

And the lines with the timestamp ar enot within an hour.all the lines are picked.

13

Change last line to /^[[]/&&v' infile to avoid non-date lines following the first match.

I don't know why you are matching dates older than 1 hour, could you post the first line that matches again,
between

```text
 and 
```

tags so the format isn't lost.

1 Like
14

Thanks Chubler.
Yes this displayed only the lines with the time stamp.
But not from last hour. All the lines from last 4days are also displayed.
appreciate your help.

15

With this input file:

****************Start Display Current Environment **********
Host OPerating Syatem is linux, version
Java version
[8/6/13 15:10:30:000 EDT] log message
[8/8/13 15:32:47:000 EDT] log message
[8/10/13 15:55:38:000 EDT] log message
[8/12/13 16:23:59:000 EDT] log message
[8/12/13 17:12:48:000 EDT] log message
[8/12/13 18:03:39:000 EDT] log message
[8/12/13 18:08:42:000 EDT] log message
[8/12/13 18:27:04:000 EDT] log message
[8/12/13 19:16:21:000 EDT] log message
[8/12/13 19:58:21:000 EDT] log message
[8/12/13 20:38:28:000 EDT] log message
[8/12/13 21:30:13:000 EDT] log message
[8/12/13 21:55:32:000 EDT] log message
[8/12/13 22:47:38:000 EDT] log message
[8/12/13 22:53:42:000 EDT] log message
[8/12/13 23:07:12:000 EDT] log message

I get this:

[8/12/13 22:47:38:000 EDT] log message
[8/12/13 22:53:42:000 EDT] log message
[8/12/13 23:07:12:000 EDT] log message

Please try you code against this file.

1 Like
16

OK can you explain the code in clear that you used. So that i will try with the original log file i have.

Thanks for the help.

17 
v=$(date --date "-1 hour" +"-vY=%-y -vT=%-m -vD=%-d -vH=%-H -vM=%-M")

This builds a string with current time - 1 hour eg:

-vY=13 -vT=8 -vD=12 -vH=23 -vM=4

This is used as a command line options to awk to set the value of 5 variables Y,T,D,H and M for Year, monTh, Day, Hour, Minute.

-F '[[ /:]' set field seperator to one of [ / space and :

Now input lines are split up into fields so from [8/12/13 23:07:12:000 EDT] log message
we get $1= $2=8 $3=12 $4=13 $5=23 $6=07 $7=12

Below will set variable v if date/time is less than 1 hour ago (i.e. year is greater than current year OR year is current year and month is greater than current month OR etc)

  $4+0>Y ||
  $4+0==Y&&$2+0>T ||
  $4+0==Y&&$2+0==T&&$3+0>D ||
  $4+0==Y&&$2+0==T&&$3+0==D&&$5+0>H ||
  $4+0==Y&&$2+0==T&&$3+0==D&&$5+0==H&&$6+0>M {v=1}

And finally print line if line starts with [ and variable v is set:
/^[[]/&&v

1 Like