parsing multi-date text file

Shell Programming and Scripting
1

Hi all:

Trying to parse a log file of rsync activity to get the amount of date being transferred. The log file contains multiple dates and what I am trying to do is get the file sizes for the current date.

What I have been trying to do is pipe it through awk but I am having trouble retrieving the output.

What I am wanting, the Total file size: for the current date. So for today, it will parse out the rsync entries for today (date +%A %Y/%b/%d) get the file size and add each for a total.

I have been trying to use the BEGIN and set the RS = "=". No joy. Is there a better way? Or am I on right track but my awk skills are insufficient?

Many thanks!

Example log file entries below:

===================================
Saturday  2010/Jun/12  16:01:24
INPUT=/incoming/check_bro
INFILE=incoming
OUTPUT=/incoming/check_bro
DEST=/incoming
SRC=/incoming/check_bro

Virus check OK

receiving file list ... done

Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 13
Total bytes sent: 8
Total bytes received: 21

sent 8 bytes  received 21 bytes  58.00 bytes/sec
total size is 0  speedup is 0.00

===================================
Saturday  2010/Jun/12  16:17:12
INPUT=/incoming/collect-CX0825BVA00141-2010-06-12-05-06-38.zip
INFILE=incoming
OUTPUT=/incoming/collect-CX0825BVA00141-2010-06-12-05-06-38.zip
DEST=/incoming
SRC=/incoming/collect-CX0825BVA00141-2010-06-12-05-06-38.zip

Virus check OK

receiving file list ... done
collect-CX0825BVA00141-2010-06-12-05-06-38.zip

Number of files: 1
Number of files transferred: 1
Total file size: 85776271 bytes
Total transferred file size: 85776271 bytes
Literal data: 85776271 bytes
Matched data: 0 bytes
File list size: 127
File list generation time: 0.332 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 42
Total bytes received: 85786932

sent 42 bytes  received 85786932 bytes  24510564.00 bytes/sec
total size is 85776271  speedup is 1.00

===================================
Thursday  2010/Jul/29  09:08:30
INPUT=/incoming/state_farm/DXiStats-CX0934BVA00518-20100729-070024.tar.gz
INFILE=incoming
OUTPUT=/incoming/state_farm/DXiStats-CX0934BVA00518-20100729-070024.tar.gz
DEST=/incoming/state_farm
SRC=/incoming/state_farm/DXiStats-CX0934BVA00518-20100729-070024.tar.gz

Virus check OK

receiving file list ... done
DXiStats-CX0934BVA00518-20100729-070024.tar.gz

Number of files: 1
Number of files transferred: 1
Total file size: 96257450 bytes
Total transferred file size: 96257450 bytes
Literal data: 96257450 bytes
Matched data: 0 bytes
File list size: 127
File list generation time: 0.369 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 42
Total bytes received: 96269391

sent 42 bytes  received 96269391 bytes  27505552.29 bytes/sec
total size is 96257450  speedup is 1.00

===================================
Thursday  2010/Jul/29  09:28:50
INPUT=/incoming/collect-CX0851BVA00415-2010-07-29-09-42-05.zip
INFILE=incoming
OUTPUT=/incoming/collect-CX0851BVA00415-2010-07-29-09-42-05.zip
DEST=/incoming
SRC=/incoming/collect-CX0851BVA00415-2010-07-29-09-42-05.zip

Virus check OK

receiving file list ... done
collect-CX0851BVA00415-2010-07-29-09-42-05.zip

Number of files: 1
Number of files transferred: 1
Total file size: 227957285 bytes
Total transferred file size: 227957285 bytes
Literal data: 227957285 bytes
Matched data: 0 bytes
File list size: 127
File list generation time: 0.854 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 42
Total bytes received: 227985302

sent 42 bytes  received 227985302 bytes  30398045.87 bytes/sec
total size is 227957285  speedup is 1.00
2

Hi,

Try this.
filesize.pl

#!/usr/bin/perl

$/="===================================";
$curr_date=`date '+%A %Y\\/%b\\/%d'`;
chop ($curr_date);
while (<>) {
if (/$curr_date(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.+?):\s(.+?)\sbytes/){
$total = $total + ${17};
}
}
print "Total file size ",$total," Bytes for today"

perl filesize.pl inputfile
1 Like