Merge values from multiple directories into one file in awk or bash

Shell Programming and Scripting
1

I am trying to merge or combine all $1 values in validation.txt from multiple directories into one new file and output it here tab-delimited: /home/cmccabe/Desktop/20x/total/total.txt . Each $2 value and the header would then be a new field in total.txt . I am not sure how to go about this as cat is not the right tool. My real data has 15 paths and 70 .txt files. Thank you :).

/home/cmccabe/Desktop/20x/panel/cardiomyopathy/coverage/validation.txt path1

Sample    20xCARDIOMYOPATHY
NA00449    98.8
NA02782    98.9

/home/cmccabe/Desktop/20x/panel/idp/coverage/validation.txt path2

Sample    20xIDP
NA00449    98.6
NA02782    98.7

/home/cmccabe/Desktop/20x/panel/pid/coverage/validation.txt path3

Sample    20xPID
NA00449    98.9
NA02782    99.0

desired output of total.txt (tab-delimited)

Sample    20xCARDIOMYOPATHY 20xIDP 20xPID
NA00449    98.8              98.6   98.9
NA02782    98.9              98.7   99.0
2

Hello cmccabe,

Could you please try following and let me know if this helps you.

awk 'BEGIN{print "Sample 20xCARDIOMYOPATHY 20xIDP 20xPID"} FNR==NR && FNR>1{A[$1]=$2;next} ($1 in A){A[$1]=A[$1]?A[$1] OFS $2:$2} END{for(i in A){if(num=split(A,X," ")>1){print i,A}}}'  Input_file1  Input_file2  Input_file3

Output will be as follows.

Sample 20xCARDIOMYOPATHY 20xIDP 20xPID
NA00449 98.8 98.6 98.9
NA02782 98.9 98.7 99.0

So above will give you output into a random sequence only. Please do let me know if this helps you.

NOTE: Also in above code put path1, path2 and path3 Input_files respectively while running the code.

Thanks,
R. Singh

3

That helps thank you very much :slight_smile: