Duplicate Line Report per Section

Shell Programming and Scripting
1

I've been working on a script (/bin/sh) in which I have requested and received help here (in which I am very grateful for!). The client has modified their requirements (a tad), so without messing up the script to much, I come once again for assistance.

Here are the file.dat contents:

ABC1 012345 header
ABC2 7890-000
ABC3 012345 Header Table  <= Need this line in report
ABC4
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580    <=Duplicate to be put in file.out
ABC5 593.5000 587.6580
ABC5 594.0000 588.0971
ABC5 594.5000 588.5361
ABC1 67890 header
ABC2 1234-0001
ABC3 67890 Header Table   <= Need this line in report
ABC4
ABC5 594.5000 588.5361 
ABC5 601.0000 594.1603
ABC5 601.5000 594.6121
ABC5 602.0000 595.0642
ABC5 602.0000 595.0642   <=Duplicate to be put in file.out

My current code will find the section header (ABC1) and the duplicates (ABC5) in that section and output that information into another file.

New client requirement: I need to add the �ABC3� line into the report

Needed Output file (file.out):

ABC1 012345 header
ABC3 012345 Header Table  <= Need this line added per section
ABC5 593.5000 587.6580 
ABC1 67890 header
ABC3 67890 Header Table   <= Need this line added per section
ABC5 602.0000 595.0642

Here is my current code:

# This will find the start of a section ABC1 and print the 
# header and duplicate lines data into a file.out
awk 'NF==3 && /ABC1/; $0!=s{s=$0;next}1' file.dat > file.out

Any suggestions?

2 
awk '/ABC[13]/&&h=$2;_[h,$2,$3]++==1' file.dat
3

That Worked!!!!!

Thank you "radoulov" for helping me out with this!

Could you explain this part?

"h=$2_[h,$2,$3]"

Thanks!

4

Yes,
but it's:

h=$2;_[h,$2,$3]++==1

not:

h=$2_[h,$2,$3]++==1

The statement separator ; matters.

There are two expressions in the code,
the first one is:

/ABC[13]/ && h=$2

If the current record matches the pattern ABC[13] AND the value returned by the assignment of the value of the second field to the variable h is true - print the record.
OK, I'm cheating here, it's a shortcut that will fail if the value of $2 is 0, on the other hand, if that value could never be 0, there's less code to write (the fact is that we don't care what's the return value of the assignment, we just need to print those records and save the value of $2).

So we print the records ABC1, ABC3 and save the numeric part of the header in the h variable.

The second expression is:

_[h,$2,$3]++==1

We build an associative array keyed by the concatenated values of the previously saved header number, the second and the third field. The values are an auto-incremented integers.
When the value is 1, we print the record, because we see it for the second time (the first time we see a key, the value is 0 (because of the post-incrementing operator).

Hope this helps.

5

Yes - Thanks for the explanation. It does help to know what I'm putting in the script. Thanks again!