awk match and merge with 2 files

Shell Programming and Scripting
1

Dear Awk experts!

I have been trying to get a one liner for a match and merge operation, but having difficulties as I'm an awk newb. I always get stuck on the match and merge with 2 files.

I have 2 files as follows:

File 1:

field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1
2,second,yes,this,1,b@c.com,n,22222,test 2
3,third,no,that,1,a@example.com,n,44444,test 4
4,fourth,no,ok,1,b@example.com,f,57657,test 3

File 2: (field separator is ":" in this file)

field 1:field 2
12345:additional1
44444:additional2
98777:additional3

So field 8 in file1 is matched with field 1 in file2.

If there is a match, then field 2 from file2 is appended to the end of the row in file 1.

So that the output looks like this:

field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1,additional1
2,second,yes,this,1,b@c.com,n,22222,test 2
3,third,no,that,1,a@example.com,n,44444,test 4,additional2
4,fourth,no,ok,1,b@example.com,f,57657,test 3

If someone could provide a breakdown of the awk statement solution, this would also be very helpful for next time!

Thanks!

2

First of all I don't see any matching values in field 7. So I guess you want to match field 8 instead of field 7.

Also your final output does not make any sense!

field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1,additional1
2,second,yes,this,1,b@c.com,n,44444,test 2,
3,third,no,that,1,a@example.com,n,76543,test 4
4,fourth,no,ok,1,b@example.com,f,44444,test 3,additional2

How the highlighted value changed from original input?

1 Like
3

Thanks for pointing that out, Yoda. My bad - I fixed it up.

4

An awk approach:

awk -F'[:,]' '
        NR == FNR {
                if ( NR > 1 )
                        A[$1] = $2
                next
        }
        {
                if ( FNR == 1 )
                {
                        print
                        next
                }
                if ( FNR > 1 && A[$8] )
                        $0 = $0 "," A[$8]
                print
        }
' file2 file1
1 Like
5 
awk -F"[:,]" 'NR==FNR{a[$1]=$2;next}
a[$8]{$0=$0","a[$8]}1' File2 File1
6

There is another inconsistency in the requested output:

field 1,field 2,field 3,field 4,field 5,field 6,field 7,field 8,field 9
1,first,yes,blah,1,a@b.com,m,12345,test 1,additional1
2,second,yes,this,1,b@c.com,n,22222,test 2,
3,third,no,that,1,a@example.com,n,44444,test 4,additional2
4,fourth,no,ok,1,b@example.com,f,57657,test 3

Note that a comma was added to the end of the line with 22222 in field 8, but no comma was added to the line with 57657 in field 8 (and neither of those values appear in field 1 in File 2).

If there is no match in File 2, should that input line be unchanged or should an empty field be added? And, should a new field have also been added to the header line in the output file?

Note that Yoda's script assumes that unmatched lines should not be changed and does not change the header line, while pravin27's script assumes an empty field should be added and adds a comma (but not a field name) to the header line.

7 
awk '
NR==FNR { # first file
  a[$1]=$2 # store in array a, hash-indexed by $1
  next # done first file
} # now remaining file
{ # print, append a[$8] if it exists
  print ($8 in a) ? $0 FS a[$8] : $0
}' FS=":" File2 FS="," File1
1 Like