awk variable search and line count between variable-search pattern

Shell Programming and Scripting
1

Input:

|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016
|Running the Rsync|Sun Oct 16 22:54:01 BST 2016
|End of the Rsync|Sun Oct 16 22:55:45 BST 2016
|Running the Rsync|Sun Oct 16 23:00:02 BST 2016
|End of the Rsync|Sun Oct 16 23:01:44 BST 2016
|Running the Rsync|Sun Oct 16 23:30:01 BST 2016
|End of the Rsync|Sun Oct 16 23:34:40 BST 2016
|Running the Rsync|Mon Oct 17 00:00:04 BST 2016
|End of the Rsync|Mon Oct 17 00:24:38 BST 2016

Every single run of the Rsync is having a number of files being transferred during the session.

I have successfully extracted time duration of the Rsync duration, but I can't understand how to calculate number of files being transferred during this one particular ( among many ).

Current, output of the script is like :

|    0  | Sun Nov 20 00:00:01 GMT 2016   | Sun Nov 20 00:02:48 GMT 2016   | 2.783333    |
|    1  | Sun Nov 20 00:30:01 GMT 2016   | Sun Nov 20 00:31:53 GMT 2016   | 1.866667    |
|    2  | Sun Nov 20 01:00:01 GMT 2016   | Sun Nov 20 01:02:24 GMT 2016   | 2.383333    |
|    3  | Sun Nov 20 01:30:01 GMT 2016   | Sun Nov 20 01:32:33 GMT 2016   | 2.533333    |
|    4  | Sun Nov 20 02:00:01 GMT 2016   | Sun Nov 20 02:00:36 GMT 2016   | 0.583333    |
|    5  | Sun Nov 20 02:30:01 GMT 2016   | Sun Nov 20 02:32:15 GMT 2016   | 2.233333    |
|    6  | Sun Nov 20 03:00:01 GMT 2016   | Sun Nov 20 03:00:34 GMT 2016   | 0.550000    |
|    7  | Sun Nov 20 03:30:01 GMT 2016   | Sun Nov 20 03:32:05 GMT 2016   | 2.066667    |
|    8  | Sun Nov 20 04:00:01 GMT 2016   | Sun Nov 20 04:00:34 GMT 2016   | 0.550000    |
|    9  | Sun Nov 20 04:30:01 GMT 2016   | Sun Nov 20 04:31:59 GMT 2016   | 1.966667    |
|   10  | Sun Nov 20 05:00:02 GMT 2016   | Sun Nov 20 05:00:35 GMT 2016   | 0.550000    |

What I'm looking for is the count of files during these Rsync sessions.

If I take a look at the variable search, I found it is impossible for me to make a variable based search.

awk -F"|" ' BEGIN { SP="Nov 22"; FP="Running the Rsync * "SP; EP="End of the Rsync * "SP; }
( $0 ~ FP ) { ST=$3;}
($0 ~ EP ) { ET=$3; TT= /* calculate time difference between these two instances here"; }
/FP/,/EP/ { x++ }

{ print "Total Time taken durng %s and %s, is %f and Number of files transfered are %d", FP, EP, TT, x }

' /var/slash/input

any help!

2

Scripts aren't very good at extracting information that is not present in the inputs that are provided. If you were to look at the output produced by rsync instead of looking at the messages your script is producing before it runs rsync and after rsync completes, you might be able to accurately count the number of successful and the number of failed attempts to transfer a file. But when that information is not available, there is no way to make it magically appear out of the ether.

3

@Don,

Between those two starting and ending lines there are thousands of lines stating the file transfer like normal Rsync does. I just skipped them because it might have filled the whole page.

below is the output.

|Running the Rsync|Sun Oct 16 22:48:01 BST 2016
+ /usr/bin/rsync -P -avz --delete  rootbox:/var/www/images /var/www/images/
receiving incremental file list
images/a55f95e69ed87c1b6fc1/large/
images/a55f95e69ed87c1b6fc1/large/1320.jpg
           0   0%    0.00kB/s    0:00:00
     1991058 100%   12.33MB/s    0:00:00 (xfer#1, to-check=1965/35841)
images/a55f95e69ed87c1b6fc1/large/1321.jpg
           0   0%    0.00kB/s    0:00:00
     1688954 100%    5.37MB/s    0:00:00 (xfer#2, to-check=1964/35841)
images/a55f95e69ed87c1b6fc1/large/1322.jpg
           0   0%    0.00kB/s    0:00:00
     1590928 100%    3.46MB/s    0:00:00 (xfer#3, to-check=1963/35841)
images/a55f95e69ed87c1b6fc1/large/1323.jpg
           0   0%    0.00kB/s    0:00:00
     1701678 100%    2.77MB/s    0:00:00 (xfer#4, to-check=1962/35841)
images/a55f95e69ed87c1b6fc1/large/1324.jpg
           0   0%    0.00kB/s    0:00:00
     1596974 100%    2.11MB/s    0:00:00 (xfer#5, to-check=1961/35841)
images/a55f95e69ed87c1b6fc1/large/1325.jpg
           0   0%    0.00kB/s    0:00:00
|End of the Rsync|Sun Oct 16 22:49:54 BST 2016
4

With that data, a file copy count could be implemented much easier...
Count the number of appearences of the string "100%", or use the xfer#nn field. What about the sixth file copy action? Was it interrupted? incomplete? Is the log file corrupt?

5

xfer#xx is only option that I am sure will be accounted for here to check the file transfer count.