Complex transpose awk script

Shell Programming and Scripting
21

Hello RudiC,

Thank you for your help!

I've done some testings to see what is inside cmpAr working only with tobe file. Now I'm trying to do the comparison loop, I got error with tmp using not as
matrix, then I change the second tmp to tmp1 but the output is not correct, I get more than 2300 fields for the sample I've posted (see the output attached).

I've done this:

awk 'BEGIN {split("OCTDP TCTDP GPRSTDP",TDPAr);
            print "MSISDN,OCTDP,TCTDP,GPRSTDP,GCSO,MCSO,SSLO,GC2SO,MC2SO,TIF,GPRSSO,OSMSSO,GC3SO,MC3SO,GC4SO,MC4SO,MMSO,STAT"
     }

    NR==FNR     {cnt[NR] = split ($0, tmp, ","); for (i=1;i<=cnt[NR]; i++) cmpAr[NR,i]=tmp; Cn++; next}
     
     /^MSISDN/   {getline; rec=$0}
     /^TDPTYPE/  {noflds=0
                  while (1)
                     {getline; if ($0 ~ /^$/) break
                      OFS="|"; tmp1=$1; $1=""; sub("\|",""); tmpAr[tmp1]=$0; noflds=3 ;
                     }
                  for (i=1; i<=noflds; i++) rec=rec","tmpAr[TDPAr]
                 }
     /^(GCSO|GC3SO|MMSO)/ {
                  getline; OFS=","; $1=$1; rec=rec","$0}
     /^NONE/     {getline; rec=rec ",no values"}
     /END/       {split (rec, tmpAr, ","); rec=tmpAr[1]
                  for (i=1;i<=Cn;i++) for (j=2;j<=cnt;j++) {if (tmpAr != cmpAr[i,j]) {tmpAr=tmpAr"_I"; dif=1} rec=rec","tmpAr}
                  if (dif) rec=rec",different"
                  print rec; rec=""; delete tmpAr; dif=0}
    ' tobe file

Almost sure that the problem is they way I'm doing the comparison loop, but I can't see the problem.

Any help would be very appreciated.

Regards

22

The comparison loop is the problem, you are right. We need to switch from testing field != fldtobe to testing !(field == fldstobe[n]) plus additional field operations. This might point you in the right direction, but it is untested, so take it as a starting point:

awk 'BEGIN {split("OCTDP TCTDP GPRSTDP",TDPAr);
            print "MSISDN,OCTDP,TCTDP,GPRSTDP,GCSO,MCSO,SSLO,GC2SO,MC2SO,TIF,GPRSSO,OSMSSO,GC3SO,MC3SO,GC4SO,MC4SO,MMSO,STAT"
     }

    NR==FNR     {fldcnt[NR] = split ($0, tmp, ","); for (i=1;i<=fldcnt[NR]; i++) cmpAr[NR,i]=tmp; rowcnt=NR; next}
     
     /^MSISDN/   {getline; rec=$0}
     /^TDPTYPE/  {noflds=0
                  while (1)
                     {getline; if ($0 ~ /^$/) break   
                      OFS="|"; tmp1=$1; $1=""; sub("\|",""); tmpAr[tmp1]=$0; noflds=3 ;
                     }
                  for (i=1; i<=noflds; i++) rec=rec","tmpAr[TDPAr]
                 }
     /^(GCSO|GC3SO|MMSO)/ {
                  getline; OFS=","; $1=$1; rec=rec","$0}
     /^NONE/     {getline; rec=rec ",no values"}
     /END/       {split (rec, tmpAr, ","); rec=tmpAr[1]
                  dif=0
                  for (fld=2; fld <= fldcnt; fld++)                        # compare fieldwise, each field on its own against all its tobe values
                    {flddif = 1                                               # assume it's different until proven wrong
                     for (row=1; row <= rowcnt; row++)                        # here we go: check against individual tobe values
                        if (tmpAr[fld] == cmpAr[row,fld]) {flddif=0; break}   # found one! Equality proven, register Equ. in flddif and break out of loop
                     if (flddif) {tmpAr[fld]=tmpAr[fld]"_I"; dif=1}           # add "_I" to field if no match found, register in dif
                     rec=rec","tmpAr[fld]}                                    # add (modified?) field to output record
                  if (dif) rec=rec",different"
                  print rec; rec=""; delete tmpAr; dif=0}
    ' tobe file

I renamed some of the variables to somewhat more mnemonic form so as not to lose track when reading. Give it a shot and report back!

23

Helllo RudicC,

Many thanks, I've tested and it seems it works just great.

I'll try it with a real file that have millions of lines.

Many thanks again for the patience and effort to help me, very appeciated. :b:

Best regards