Stack data from multiple files into one, with variable column files

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hello Gurus,
Im new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldnt find a solution to my issue. Greatly appreciate any help..

I have three csv files -> Apex_10_Latest.csv , Apex_20_Latest.csv , Apex_30_Latest.csv . Number of columns are varying in these 3 files. Typically the latest file, based on the numbering, might have some new columns appended to the end. So I want to take the latest header and stack the data from all the 3 files into a new file Apex.csv . When stacking the data from older file which might have less columns than latest file, I want the data to be populated as null with appropriate delimiters..

Also this has to be done recursively for a multiple set of files (3 each), all in the same folder.

  • Apex_10_Latest.csv , Apex_20_Latest.csv , Apex_30_Latest.csv - merged into Apex.csv
  • Code_10_Latest.csv , Code_20_Latest.csv , Code_30_Latest.csv - merged into Code.csv
  • Trans_10_Latest.csv , Trans_20_Latest.csv , Trans_30_Latest.csv - merged into Trans.csv

Following is the format of the source files and expected target file...
SOURCE FILES:

  • Apex_30_Latest.csv :
A,B,C,D 
----------- 
1,2,3,4 
2,3,4,5 
3,4,5,6
  • Apex_20_Latest.csv :
A,B,C 
--------- 
4,5,6 
5,6,7 
6,7,8
  • Apex_10_Latest.csv
A,B
------- 
7,8
8,9 
9,10

EXPECTED TARGET FILE:

  • Apex.csv
A,B,C,D
----------
1,2,3,4
2,3,4,5
3,4,5,6
4,5,6,,
5,6,7,,
6,7,8,,
7,8,,,
8,9,,,
9,10,,,

Thanks...

2

Please use code tags as required by forum rules!

Any attempts from your side?

---------- Post updated at 11:24 ---------- Previous update was at 11:13 ----------

Your desired output has one delimiter too many in the second and third file, or one missing i the first.
The problem is the reverse order of the input files. Try

awk 'NR==1 {MAXNF=NF} NR<=2; FNR <=2 {next} {for (i=NF+1; i<=MAXNF; i++) $i=""} 1' FS="," OFS="," $(ls -r Ape*)
A,B,C,D
-----------
1,2,3,4
2,3,4,5
3,4,5,6
4,5,6,
5,6,7,
6,7,8,
7,8,,
8,9,,
9,10,,