Generating formatted reports from log files

Shell Programming and Scripting
1

Given that I have a log file of the format:

DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ
etc

How do I produce a report that looks like

ID = T-4030097550
2009-07-23T14:05:11Z MessX
2009-07-23T14:10:44Z MessY
2009-07-23T15:05:10Z MessZ
ID = T-7298651656
2009-07-23T14:34:08Z MessX
2009-07-23T14:41:00Z MessY
etc

Other info:
The IDs are not pre-determined
The messages in the report are actually a subset of the log-entries and some messages may be missing for a given ID.

2

Try this,give your file name as argument to script

#!/bin/sh
for id in `cut -d' ' -f2 $1 |sort -u`
do
      echo 'ID='$id
      grep $id $1 |awk '{print $1 " "  $5}'
done
3

That works well.

One thing I ommitted was that the message elements is actually a text string so each record is more of the format

2009-07-27T08:08:18Z T-6769544753 D | this is some log entry

In this case I wnat to pick up all fields from column 5

The other thing is if I only want to pick up log entries that contain strings of interest is there a simple of way of seeding this from a file with the entries.

My alternative was to run a log of grep statements for each string and then sort the resulting file by timestamp (column 1) before presenting it to the script you provided. - I suspect there is a better way.

4

replace

 grep $id $1 |awk '{print $1 " "  $5}'

with

grep $id $1 |sed 's/ .*|//'
5

perl:

while(<DATA>){
	chomp;
	next if $. == 1;
  my @tmp=split("[ |]",$_);
  push @{$hash{$tmp[1]}}, $tmp[0]." ".$tmp[5];
}
foreach my $key(keys %hash){
	print "ID = $key\n";
	print join "\n", @{$hash{$key}};
	print "\n";
}
__DATA__
DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ
6

I think the code by JohnBach is good enough:

#!/bin/sh
lookfor='(MessX|MessY|MessZ|this|that|blabla)'
for id in `cut -d' ' -f2 $1 |sort -u` ; do
      echo 'ID='$id
      grep $id $1 | grep $lookfor |awk '{print $1 " "  $5}'
done