Using awk with the date command and escape characters

Shell Programming and Scripting
1

I have a file that is a log file for web traffic. I would like to convert the timestamp in it to unix time or epoch time.

I am using the date command in conjunction with awk to try to do this. Just

myfile:

28/Aug/1995:00:00:38 1 /pub/atomicbk/catalog/home.gif 813
28/Aug/1995:00:00:38 1 /pub/atomicbk/catalog/catalog.gif 693
28/Aug/1995:00:00:39 1 /pub/atomicbk/catalog/laser.gif 129
28/Aug/1995:00:00:40 1 /pub/atomicbk/catalog/logo2.gif 12871

I can run the date command like this:

date -j -f "%d/%b/%Y:%T" 28/Aug/1995:00:00:38 +%s

And I get the intended result:

809593238

(809593230 seconds since Jan 1, 1970)

So myfile above, the expected output should be:

809593238 1 /pub/atomicbk/catalog/home.gif 813
809593238 1 /pub/atomicbk/catalog/catalog.gif 693
809593239 1 /pub/atomicbk/catalog/laser.gif 129
809593240 1 /pub/atomicbk/catalog/logo2.gif 12871

I'm close but I'm getting an error.

I run the following command:

cat myfile | awk '{ date -j -f "%d/%b/%Y:%T" print $1 echo +%s }'

The error I get is:

awk: syntax error at source line 1
 context is
    { date -j -f "%d/%b/%Y:%T" >>>  print <<<  $1 echo " +%s" }
awk: illegal statement at source line 1

I've tried various combinations of this but with no success. I think it could be because 1) commands like date may not work inside an awk statement and 2) I need to properly escape the '+' and the '%' inside an awk statement so it can be passed on to the date command correctly. Any help is much appreciated.

Thanks!

2

You need a bit more code to make it work with awk :).

Anyway, I would use Perl:

perl -MTime::Local -lane'BEGIN {
  $month{$_} = ++ $c 
    for "JANFEBMARAPRMAYJUNJULAUGSEPOCTNOVDEC" =~ /.../g
  }
  @dt = split /[\/:]/, $F[0];
  $F[0] = timelocal $dt[5], $dt[4], $dt[3], 
    $dt[0], $month{uc $dt[1]} - 1 , $dt[2] - 1900;
  print join " ", @F;
  ' infile

This a quick and dirty solution, there are non-standard modules on CPAN that will parse the date for you and your code will be more robust.

3

Or shell:

while read fdate rest; do 
  echo $(date -j -f "%d/%b/%Y:%T" $fdate +%s) "$rest"
done < myfile
4

Thank you so much!

Both the perl and the while read solution worked great! I'll let you know if one runs faster than the other as I will be running this on approximately 1 million lines.

---------- Post updated at 01:02 PM ---------- Previous update was at 08:49 AM ----------

Ok, so I ran both the Perl and the "while read" code blocks on a 309MB log file that had 6,397,194 lines. While both code blocks performed the same end result, the Perl block of code performed exponentially faster, so I had to use that instead of the while read. I waited about 10 minutes on the "while read" block and it was still going. The Perl block finished under 3 minutes. So I ended up using the Perl block in my script.

Thank you both for your help!