filtering the logs

Shell Programming and Scripting
1

Hi,

We are using rsync for syncing remote directories. It is working great along with detailed logs. As the script cron'd and most of the times there're no files to sync we are getting lot of unnecessary log entries and we need to filter them to show only the log entries for the files transferred.

Actual Log: (part of)

2008/04/08 21:43:17 [19631] .d..t.... 09/
2008/04/08 21:43:17 [19631] sent 51 bytes received 26 bytes 1.86 bytes/sec
2008/04/08 21:43:17 [19631] total size is 0 speedup is 0.00
2008/04/08 21:44:49 [5672] building file list
2008/04/08 21:44:49 [5672] done
2008/04/08 21:44:49 [5672] .d..t.... 09/
2008/04/08 21:44:49 [5672] sent 51 bytes received 26 bytes 1.86 bytes/sec
2008/04/08 21:44:49 [5672] total size is 0 speedup is 0.00
2008/04/08 21:46:49 [31047] building file list
2008/04/08 21:46:49 [31047] done
2008/04/08 21:46:49 [31047] .d..t.... 09/
2008/04/08 21:46:49 [31047] <f+++++++ 09/file1.pdf
2008/04/08 21:46:49 [31047] sent 110 bytes received 48 bytes 3.81 bytes/sec
2008/04/08 21:46:49 [31047] total size is 0 speedup is 0.00
2008/04/08 21:48:49 [25274] building file list
2008/04/08 21:48:49 [25274] done
2008/04/08 21:48:49 [25274] .d..t.... 09/
2008/04/08 21:48:49 [25274] sent 71 bytes received 26 bytes 2.40 bytes/sec
2008/04/08 21:48:49 [25274] total size is 0 speedup is 0.00
2008/04/08 21:50:49 [18829] building file list
2008/04/08 21:50:49 [18829] done
2008/04/08 21:50:49 [18829] .d..t.... 09/
2008/04/08 21:50:49 [18829] sent 71 bytes received 26 bytes 2.34 bytes/sec

I achieved the below by using grep:

2008/04/08 21:43:17
2008/04/08 21:43:17 09/
2008/04/08 21:44:49
2008/04/08 21:44:49 09/
2008/04/08 21:46:49
2008/04/08 21:46:49 09/
2008/04/08 21:46:49 09/file1.pdf
2008/04/08 21:48:49
2008/04/08 21:48:49 09/
2008/04/08 21:50:49
2008/04/08 21:50:49 09/
2008/04/08 21:52:49

I need to get only the entries that have a filename (that was actually transferred) as below:

2008/04/08 21:46:49 09/file1.pdf

Thanks
Prvn

2

This may work for you if all the filenames being rsynced have the normal three character extension like in your example (file1.pdf):

grep -e '\.[A-Za-z0-9]\{3\}' log

This will allow grep to match anything with a '.' followed by a three character file extension.

Hope this helps.

3 
awk '$3 =~ /\/./'

Might as well run that on the original log, in which case the field would seem to be $5

4

Thank you for your replies.

files may or may not contain extensions. i found solution for this (sorry if did not provide you enough info.) and it is

date1="09"
grep "$date1/" 1.txt |grep -v "$date1/"$

Era - i am getting this below error with your solution.

-sh-3.1# cat 1.txt |awk '$3 =~ /\/./'
awk: $3 =~ /\/./
awk: ^ syntax error
-sh-3.1# cat 1.txt |gawk '$3 =~ /\/./'
gawk: $3 =~ /\/./
gawk: ^ syntax error

Thanks
Prvn

5

Sorry, mixed in some Perl syntax there. Take out the =, it should be just ~ (and take out the Useless Use of Cat too -- google that).

6 

awk '
/09\/$/, /done/ {
  for ( i=1;i<=NF;i++ ) {
   if ( $i ~ /<f+++++++/ ) {
      sub("09/","",$(i+1)) 
      print $(i+1)
   }
  }
}' file