merging CSV data using a one liner from shell?

Shell Programming and Scripting
1

I'm trying to merge multiple CSV (comma separated value) files into one large master file. All files have a field that is unique to act as the key for entry/merging into the master file & and all files have the same number of fields that are in the master file.

I'll give an example here:

MASTERFILE (BEFORE MERGE):

"userid","logon","name","fname","lname","app1","app2","app3","app4","app5"
"111","","Joe One","","","yes","no","","no",""
"333","","","","","","","","",""
"555","","","Five","","","","","",""
"222","mrtwo","Lou Two","Lou","","no","no","no","no","yes"

MERGEFILE1:

"111","mrone","","Joe","One","","","yes","","no"
"222","","Lou Two","Lou","","no","no","no","no","yes"
"555","mrfive","","","","","","","",""

I imagine it would be something like:
Run the script...
# {somescript} masterfile mergefile1 > masterfile2

So now MASTERFILE2 would become:

"111","mrone","Joe One","Joe","One","yes","no","yes","no","no"
"222","mrtwo","Lou Two","Lou","","no","no","no","no","yes"
"333","","","","","","","","",""
"555","mrfive","","Five","","","","","",""

So the first field would be be scanned from the first line of the mergefile to see if it is in the the first filed of any lines in the masterfile, if it exists in the masterfile, then it would update the fields for that entry accordingly (overwriting any existing fields in masterfile) and continue on to process the next line of the mergefile against the masterfile.

How could I accomplish this? :confused:

Thanks!

2

The task breaks down to this.

drop the header lines as needed.

insert a second column (sed will work) to both files to indicate the file number (1 or 2 - in your example)

sort the two files together

an awk program to merge the two files together and output the results.

does that make sense? which step to you need help on?

3

Hmm... aside from, drop the header lines as needed... all of them?