Concatenate select lines from multiple files

Shell Programming and Scripting
1

I have about 6000 files of the following format (three simplified examples shown; actual files have variable numbers of columns, but the same number of lines). I would like to concatenate the ID (*Loc*) and data lines, but not the others, as shown below. The result would be one large file (or several slightly less large files).

fileA

header line A
A_Loc1, A_Loc2, A_Loc3, A_Loc4
Pop
gt , 02 04 01 01 
gt , 02 04 01 01 
Pop
gt , 03 04 01 01 
gt , 03 04 01 01

fileB

header line B
B_Loc1, B_Loc2
Pop
gt , 03 03
gt , 02 03
Pop
gt , 02 03 
gt , 02 03

fileC

header line C
C_Loc1, C_Loc2, C_Loc3
Pop
gt , 03 03 02
gt , 03 03 02
Pop
gt , 03 03 02
gt , 03 03 02

Output:

header line A
A_Loc1, A_Loc2, A_Loc3, A_Loc4, B_Loc1, B_Loc2, C_Loc1, C_Loc2, C_Loc3
Pop
gt , 02 04 01 01 03 03 03 03 02
gt , 02 04 01 01 02 03 03 03 02
Pop
gt , 03 04 01 01 02 03 03 03 02
gt , 03 04 01 01 02 03 03 03 02

I know paste can concatenate lines, but I'm not sure how to only paste selective lines, not how to add a comma in the ID lines. Would be grateful if someone could show me a way.

2

If "ls file*" lists only those files in the directory, then

awk 'NR == FNR {a[FNR] = $0; next} NF > 1 {if($0 !~ /^gt/) {$2 = $0}; a[FNR] = (a[FNR] $2)} END {for(i = 1; i <= FNR; i++) {print a}}' FS=',' file*
3

This is a general join on field#1:

awk '($1==k[FNR]) {$1=""} {a[FNR]=a[FNR] FS $0} (NR==FNR) {k[FNR]=$1} END {for (i=1;i<=FNR; i++) print a}' file*
4

If there are too many files and you get : "Too many arguments", you could try:

for f in file*
do
   [ -f "$f" ] || continue
   cat "$f"
done | 

awk -F, '
  /header/ {
    n=c
    c=0
  } 
  !(++c in A) {
    A[c]=$0
    next
  }
  /Loc/ {
    A[c]=A[c]", "$0
  }
  $1~/^gt/ {
    A[c]=A[c] $2
  }
  END {
    for(i=1; i<=n; i++)print A
  }
'