Rearrange groups of lines from several files

Shell Programming and Scripting
1

I have three files as an input and I need to rearrange this input to match the rules by which the processing program consumes the data.

My files are:

/tmp$ cat F[123]
# file -1-
FS00|0|zero-zero|
FSTA|0|10|
FSTA|0|12|
FSTA|0|15|
FSTA|0|17|
FS00|3|negative|
FSTA|3|-1|
FS00|5|regular|
FSTA|5|21|
#
GXRM|0|0|1|0
GXRX|120|200|220|220|
GXRX|116|202|222|222|
GXRM|2|1|0|0
GXRX|1|0|0|0|
#
STPE|Initial|
#
FNMR|20|12|11|12|11|0|
FNMR|21|15|10|10|10|5|
# file -2-
RGSS|0|1|1|1|1|0|
RGSS|1|1|1|1|1|2|
RGSS|7|6|1|9|2|2|
# file -3-
TESR|0.00|0.00|0|0|0.00|FIX
TESR|1.00|0.00|5|0|0.00|FIX CNT
TESR|4.20|1.05|5|5|0.10|FIX REV
TESR|0.00|0.00|8|7|0.00|FLEX

I need to combine them in this fashion:

STPE|Initial|
FS00|0|zero-zero|
FSTA|0|10|
FSTA|0|12|
FSTA|0|15|
FSTA|0|17|
FS00|3|negative|
FSTA|3|-1|
FS00|5|regular|
FSTA|5|21|
GXRM|0|0|1|0
GXRX|120|200|220|220|
GXRX|116|202|222|222|
GXRM|2|1|0|0
GXRX|1|0|0|0|
TESR|0.00|0.00|0|0|0.00|FIX
TESR|1.00|0.00|5|0|0.00|FIX CNT
TESR|4.20|1.05|5|5|0.10|FIX REV
TESR|0.00|0.00|8|7|0.00|FLEX
RGSS|0|1|1|1|1|0|
RGSS|1|1|1|1|1|2|
RGSS|7|6|1|9|2|2|
FNMR|20|12|11|12|11|0|
FNMR|21|15|10|10|10|5|

See, how I need groups of lines following each other, such as STPE to be 1st group, followed by FS group. Important to keep lines within each group in its original order.

I tried this awk (NOTE: gawk is not available, this is old awk) where I accumulate my groups of lines into arrays and I slap NR to be used later to re-sort output in the order these lines came in and I cut it off on the output.

/tmp$ cat f.awk
BEGIN { FS = OFS = "|" }
/^TESR/ {       TESR[NR"|"$0] = 1;      }
/^RGSS/ {       RGSS[NR"|"$0] = 1;      }
/^FNMR/ {       FNMR[NR"|"$0] = 1;      }
/^STPE/ {       STPE[NR"|"$0] = 1;      }
/^GX../ {       GX__[NR"|"$0] = 1;      }
/^FS../ {       FS__[NR"|"$0] = 1;      }
{
        continue;
}
END {
        for(ln in STPE) { print ln | "sort -n|cut -d'|' -f2-" }
        for(ln in FS__) { print ln | "sort -n|cut -d'|' -f2-" }
        for(ln in GX__) { print ln | "sort -n|cut -d'|' -f2-" }
        for(ln in TESR) { print ln | "sort -n|cut -d'|' -f2-" }
        for(ln in RGSS) { print ln | "sort -n|cut -d'|' -f2-" }
        for(ln in FNMR) { print ln | "sort -n|cut -d'|' -f2-" }
}

The output is not at all what I expect:

/tmp$ awk -f f.awk F[123]
FS00|0|zero-zero|
FSTA|0|10|
FSTA|0|12|
FSTA|0|15|
FSTA|0|17|
FS00|3|negative|
FSTA|3|-1|
FS00|5|regular|
FSTA|5|21|
GXRM|0|0|1|0
GXRX|120|200|220|220|
GXRX|116|202|222|222|
GXRM|2|1|0|0
GXRX|1|0|0|0|
STPE|Initial|
FNMR|20|12|11|12|11|0|
FNMR|21|15|10|10|10|5|
RGSS|0|1|1|1|1|0|
RGSS|1|1|1|1|1|2|
RGSS|7|6|1|9|2|2|
TESR|0.00|0.00|0|0|0.00|FIX
TESR|1.00|0.00|5|0|0.00|FIX CNT
TESR|4.20|1.05|5|5|0.10|FIX REV
TESR|0.00|0.00|8|7|0.00|FLEX

Please help, any idea will be appreciated

2

So, I added a "group id" to be included in the sort and solved it that way, and I don't pipe to sort individual arrays (which I suspect was a bug)
Just in case my code now looks like this:

/^STPE/ {       STPE["1|"NR"|"$0] = 1;  }
/^FS../ {       FS__["2|"NR"|"$0] = 1;  }
/^GX../ {       GX__["3|"NR"|"$0] = 1;  }
/^TESR/ {       TESR["4|"NR"|"$0] = 1;  }
/^RGSS/ {       RGSS["5|"NR"|"$0] = 1;  }
/^FNMR/ {       FNMR["6|"NR"|"$0] = 1;  }


...


END {
        for(ln in STPE) { print ln }
        for(ln in FS__) { print ln }
        for(ln in GX__) { print ln }
        for(ln in TESR) { print ln }
        for(ln in RGSS) { print ln }
        for(ln in FNMR) { print ln }
}

and on the output I do

 sort -t'|' -k1,1n -k2,2n | cut -d'|' -f3-

to clean out those sort-related columns

3

Why that difficult? Try

awk '
BEGIN { FS = OFS = "|" }

/^TESR/ {TESR[++CNTTESR] = $0}
/^RGSS/ {RGSS[++CNTRGSS] = $0}
/^FNMR/ {FNMR[++CNTFNMR] = $0}
/^STPE/ {STPE[++CNTSTPE] = $0}
/^GX../ {GX__[++CNTGX__] = $0}
/^FS../ {FS__[++CNTFS__] = $0}

END     {for (i=1; i<=CNTSTPE; i++) print STPE
         for (i=1; i<=CNTFS__; i++) print FS__
         for (i=1; i<=CNTGX__; i++) print GX__
         for (i=1; i<=CNTTESR; i++) print TESR
         for (i=1; i<=CNTRGSS; i++) print RGSS
         for (i=1; i<=CNTFNMR; i++) print FNMR
        }
 ' file[123]

to get exactly your desired output.

EDIT: Or even

awk '
BEGIN   {FS = OFS = "|"
         MX = split ("STPE|FS..|GX..|TESR|RGSS|FNMR", KEYS)
        }

        {for (i=1; i<=MX; i++) if ($1 ~ KEYS)        {RES[i,++CNT] = $0 
                                                         break
                                                        }
        }
END     {for (i=1; i<=MX; i++) 
           for (j=1; j<=CNT; j++) print RES[i, j]
        }
' file[123]