Extract section of file based on word in section

Shell Programming and Scripting
1

I have a list of Servers in no particular order as follows:

virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"

And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection).

9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS (Execution unit 12)
9/17/2010 8:01:49 PM: Copying IIBVICDMS01
9/17/2010 8:01:56 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01-000002-flat.vmdk" using "Network"
9/17/2010 8:06:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01_1-000002-flat.vmdk" using "Network"
9/17/2010 8:19:00 PM: Execution completed successfully

9/17/2010 8:19:00 PM: Completed: 7 files, 790.1 GB
9/17/2010 8:19:00 PM: Performance: 47446.3 MB/minute
9/17/2010 8:19:00 PM: Duration: 00:18:54 (00:01:50 idle/loading/preparing)
9/17/2010 8:00:04 PM: Normal backup using VDRBACKUPS (Execution unit 11)
9/17/2010 8:01:06 PM: Copying IIBVICMA01
9/17/2010 8:01:14 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:26:23 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:36:39 PM: Execution completed successfully

9/17/2010 8:36:39 PM: Completed: 7 files, 362.1 GB
9/17/2010 8:36:39 PM: Performance: 10461.5 MB/minute
9/17/2010 8:36:39 PM: Duration: 00:36:35 (00:01:09 idle/loading/preparing)
9/17/2010 8:00:02 PM: Normal backup using VDRBACKUPS (Execution unit 10)
9/17/2010 8:00:55 PM: Copying IIBSBS
9/17/2010 8:01:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS-flat.vmdk" using "Network"
9/17/2010 8:58:37 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_1-flat.vmdk" using "Network"
9/17/2010 9:47:28 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_2-flat.vmdk" using "Network"
9/17/2010 9:49:05 PM: Execution completed successfully

9/17/2010 9:49:05 PM: Completed: 9 files, 514.6 GB
9/17/2010 9:49:05 PM: Performance: 4879.8 MB/minute
9/17/2010 9:49:05 PM: Duration: 01:49:01 (00:01:02 idle/loading/preparing)

For each SERVERNAME in ${virtualMachines} I want to extract each section beginning with "Normal backup" and ending with "Duration" to a file.

The Server name is a constant and will always appears in the "Normal backup/Duration" block of output (typically as "Copying $SERVERNAME" but this can change if it has an error but is always there). So it's kind of like a grep of the $SERVERNAME that also grabs the "Normal backup -> Duration" section that the $SERVERNAME exists within. This also needs to append to the file as sometimes the backup of each server might run more than once and produce more than 1 "Normal backup -> Duration" section.

Another way to describe it would be...

Output only "Normal backup -> Duration" section where $SERVERNAME = IIBSBS and append output to /tmp/vdrlog-IIBSBS.txt

e.g. it would look something like this. My script is just to try and illustrate what I want and does not work.

#!/bin/bash

virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"
backupLog="/tmp/backup.log"

for i in ${virtualMachines} ; do
        grep $i ${backupLog} | awk
                '$4 == "Normal" && $5 == "backup"{f=1; c++}
                f{print >> "/tmp/vdrlog-"'"$i"'"".txt}
                $4 ~ /Duration/{f=0}'
done

cat /tmp/vdrlog-IIBSBS.txt
9/17/2010 8:00:02 PM: Normal backup using VDRBACKUPS (Execution unit 10)
9/17/2010 8:00:55 PM: Copying IIBSBS
9/17/2010 8:01:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS-flat.vmdk" using "Network"
9/17/2010 8:58:37 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_1-flat.vmdk" using "Network"
9/17/2010 9:47:28 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_2-flat.vmdk" using "Network"
9/17/2010 9:49:05 PM: Execution completed successfully

9/17/2010 9:49:05 PM: Completed: 9 files, 514.6 GB
9/17/2010 9:49:05 PM: Performance: 4879.8 MB/minute
9/17/2010 9:49:05 PM: Duration: 01:49:01 (00:01:02 idle/loading/preparing)

cat /tmp/vdrlog-IIBVICDMS01.txt
9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS (Execution unit 12)
9/17/2010 8:01:49 PM: Copying IIBVICDMS01
9/17/2010 8:01:56 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01-000002-flat.vmdk" using "Network"
9/17/2010 8:06:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01_1-000002-flat.vmdk" using "Network"
9/17/2010 8:19:00 PM: Execution completed successfully

9/17/2010 8:19:00 PM: Completed: 7 files, 790.1 GB
9/17/2010 8:19:00 PM: Performance: 47446.3 MB/minute
9/17/2010 8:19:00 PM: Duration: 00:18:54 (00:01:50 idle/loading/preparing)

cat /tmp/vdrlog-IIBVICMA01.txt
9/17/2010 8:00:04 PM: Normal backup using VDRBACKUPS (Execution unit 11)
9/17/2010 8:01:06 PM: Copying IIBVICMA01
9/17/2010 8:01:14 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:26:23 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:36:39 PM: Execution completed successfully

9/17/2010 8:36:39 PM: Completed: 7 files, 362.1 GB
9/17/2010 8:36:39 PM: Performance: 10461.5 MB/minute
9/17/2010 8:36:39 PM: Duration: 00:36:35 (00:01:09 idle/loading/preparing)

I previously had great help from Franklink52 in this thread which is what was using to grab the sections but I am updating the script so it accounts for situations where sometimes the backup will run more than once or if errors occur in the backup.

2 
#!/usr/bin/env ruby 
vmachines=%w(IIBVICDMS01 IIBVICMA01)
#vmachines=%w(IIBSBS IIBVICDMS01 IIBVICMA01)
s=""
while line=gets
  host=line.split[-1] if line =~ /Copying/
  if line.match("Duration")
    setter = 0 and s << line
    if vmachines.include?(host)
      File.open("/tmp/vdrlog-"+host+".txt","w") {|x| x.puts s }
    end
    s=""
  elsif line.match("Normal")
    setter=1
  end
  s <<line if setter
end


<your command/script that generate the output> ... | ruby myscript.rb
3

If you don't need the "Normal backup" line then this can do:

virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"
for i in ${virtualMachines}; do awk "/$i/,/Duration/" /tmp/backup.log > vdrlog-$i; done

In case that this line is important for you, let me know I'll come up with some Perl.

1 Like