Parse a file using awk

rajangupta2387 · March 8, 2013, 8:21am

Hi Experts,

I am trying to parse the following file;
FILEA

a|b|c|c|c|c
a|b|d|d|d|d
e|f|a|a|a|a
e|f|b|b|b|b

output expected:

a<TAB>b
<TAB><TAB>c<TAB>c<TAB>c<TAB>c<TAB>
<TAB><TAB>d<TAB>d<TAB>d<TAB>d<TAB>
e<TAB>f
<TAB><TAB>a<TAB>a<TAB>a<TAB>a<TAB>
<TAB><TAB>b<TAB>b<TAB>b<TAB>b<TAB>

where <TAB> is spaces.

I have tried using the this file as

 
cat FILEA |
  /usr/local/bin/gawk -F'|' '{for(i=1;i<NF;i++)
  {
   n=split($i,e,/[|]/)
   exists[e[1]]
   val[e[1],e[2],e[3],e[4],e[5]]
     }}
  END{
  no_of_names=asorti(exists)
  for(i=1;i<=no_of_names;i++)
  {
   for(j in val)
   {
    split(j,temp,SUBSEP)
    if(temp[1]==exists)
    print  k[1],temp[j+3],temp[4], val[j] | "sort -nr"
   }
   close("sort -nr")
   printf "\n"
  }
  }' OFS='|'

Please suggest what am i doing wrong

Yoda · March 8, 2013, 8:43am

sort file | awk -F'|' '
 A[1] == $1 && A[2] == $2 {
        n = split ($0, A, /[|]/)
        printf OFS
        for ( i = 3; i <= n; i++)
                printf OFS A;
        printf RS
} A[1] != $1 && A[2] != $2 {
        n = split ($0, A, /[|]/)
        print A[1], A[2]
        printf OFS
        for ( i = 3; i <= n; i++)
                printf OFS A;
        printf RS
} {
        n = split ($0, A, /[|]/)
} ' file

rajangupta2387 · March 8, 2013, 9:04am

Thanks afor your answer and it work for the situation i mentioned. But in actual the contents of the file are large in size and it fails to execute that.

actual file is something like:

 
Comptel_6_UAT|EventLinkUserHome|StreamParameterValue|Value|NOT EXIST|/u01/appl/u1elkpe
Comptel_6_UAT|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|Packages|EL_LIB_SFTP|1.1.0.|1.2.1.
Comptel_6_UAT|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|Packages|EL_CFG_FTP_DISTRIBUTOR|2.1.0.|2.1.0.upc2
Comptel_6_UAT|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|Packages|EL_APP_FTP_DISTRIBUTOR|2.1.4.|2.1.5.
Comptel_6_UAT|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TargetDirectory|/u01/appl/u1elkpe/delivery/UPC_PE_Rejected_Loader_stream/INPUT/RO|${EventLinkUserHome}/delivery/UPC_PE_Rejected_Loader_stream/INPUT/RO
Comptel_6_UAT|RO_OPEN_AUDIT_REJECTED_ENCODER|Packages|EL_LIB_CML|1.5.4.|1.5.5.
Comptel_6_UAT|RO_OPEN_BLN|Packages|UPC_LIB_PERL_UTILS|1.10.1.|3.2.1.
Comptel_6_UAT|RO_OPEN_BLN|Packages|UPC_CFG_RO_OPENSER_BLN|1.4.0.|1.6.2.
Comptel_6_UAT|RO_OPEN_BLN|NodeParameters|StreamName|${StreamName}|NOT EXIST
Comptel_6_UAT|RO_OPEN_BLN|NodeParameters|LsAdmin|/u01/appl/u1elkpe/EventLink/lookup_server3/bin/ls_admin|NOT EXIST
Comptel_6_UAT|RO_OPEN_BLN|NodeParameters|Lookup.ShmidDirectory|/u01/appl/u1elkpe/EventLink/lookup_server3/tmp|${EventLinkUserHome}/EventLink/lookup_server3/tmp
Comptel_6_UAT|RO_OPEN_BLN|NodeParameters|LD_PRELOAD|/usr/lib/64/libCstd.so.1|NOT EXIST
Comptel_6_UAT|RO_OPEN_BRIO_DISTRIBUTOR|NodeParameters|TargetDirectory|/u01/appl/u1elkpe/delivery/UPC_RO/OPENSER/OUTPUT/BRIO|NOT EXIST
Comptel_6_UAT|RO_OPEN_BRIO_ENCODER|Packages|UPC_CFG_RO_OPENSER_BRIO_ENCODER|1.1.1.|NOT EXIST
Comptel_6_UAT|RO_OPEN_DR_DISTRIBUTOR|NodeParameters|Username| |u1elkpe
Comptel_6_UAT|RO_OPEN_DR_DISTRIBUTOR|NodeParameters|TargetDirectory|/u01/appl/u1elkpe/delivery/UPC_RO/OPENSER/OUTPUT/DR|${EventLinkUserHome}/delivery/UPC_RO/OPENSER/OUTPUT/DR
Comptel_6_UAT|RO_OPEN_DR_DISTRIBUTOR|NodeParameters|Password| |u1elkpe
Comptel_6_UAT|RO_OPEN_DWH_DISTRIBUTOR|NodeParameters|Username|NOT EXIST|u1elkpe
Comptel_6_UAT|RO_OPEN_FILE_DUPLICATE_CHECKER|NodeParameters|SequenceMax|999999|999999999
Comptel_6_UAT|RO_OPEN_FILE_DUPLICATE_CHECKER|NodeParameters|NumberPosition|38|27
Comptel_6_UAT|RO_OPEN_FILE_DUPLICATE_CHECKER|NodeParameters|NumberLength|6|10
Comptel_6_UAT|RO_OPEN_FTP_COLLECTOR|Packages|EL_APP_FTP_COLLECTOR|2.1.4.upc1|2.1.6.upc1
Comptel_6_UAT|RO_OPEN_FTP_COLLECTOR|NodeParameters|SourceDirectory|/u01/appl/u1elkpe/delivery/UPC_RO/OPENSER/INPUT|${EventLinkUserHome}/delivery/UPCRO_OPENSER/INPUT
Comptel_6_UAT|RO_OPEN_KENAN_DISTRIBUTOR|NodeParameters|Username| |u1elkpe
Comptel_6_UAT|RO_OPEN_KENAN_DISTRIBUTOR|NodeParameters|TargetDirectory|/u01/appl/u1elkpe/delivery/UPC_RO/OPENSER/OUTPUT/KENAN|${EventLinkUserHome}/delivery/UPC_RO/OPENSER/OUTPUT/KENAN
Comptel_6_UAT|RO_OPEN_KENAN_ENCODER|Packages|UPC_CFG_RO_KENAN_ENCODER|1.2.1.|1.3.0.
Comptel_6_UAT|RO_OPEN_KENAN_RDC|Packages|UPC_CFG_RECORD_CHECKER|7.1.6.|NOT EXIST
Comptel_6_UAT|RO_OPEN_KENAN_RDC|NodeParameters|TimestampField|EL_CALL_A_TIME|NOT EXIST
Comptel_6_UAT|RO_OPEN_KENAN_RDC|NodeParameters|Keys|EL_TERMINATING_NUMBER,EL_ORIGINATING_NUMBER,I_ORIG_TRUNK_NUM,I_TERM_TRUNK_NUM,EL_SWITCH_ID|NOT EXIST
Comptel_6_UAT|RO_OPEN_KENAN_RDC|NodeParameters|CDR_DURATION|EL_DURATION|NOT EXIST
Comptel_6_UAT|RO_OPEN_KENAN_RDC|NodeParameters|B_NUMBER|EL_TERMINATING_NUMBER|NOT EXIST
Comptel_6_UAT|RO_OPEN_KENAN_RDC|NodeParameters|A_NUMBER|EL_ORIGINATING_NUMBER|NOT EXIST
Comptel_6_UAT|RO_OPEN_KENAN_RDC|DataStorages|UPC_CFG_RECORD_CHECKER|DUPLICATE_RECORDS|NOT EXIST
Comptel_6_UAT|RO_OPEN_OUTPUT_BACKUP|NodeParameters|TargetDirectory|/u01/appl/u1elkpe/delivery/UPC_RO/OPENSER/BACKUP/|NOT EXIST
Comptel_6_UAT|RO_OPEN_OUTPUT_BACKUP|NodeParameters|InputLinkDirectories|Yes|NOT EXIST
Comptel_6_UAT|RO_OPEN_OUTPUT_BACKUP|Links|ENCODED_AUDIT|1|NOT EXIST
Comptel_6_UAT|RO_OPEN_SWITCH_BACKUP|NodeParameters|TargetDirectory|/u01/appl/u1elkpe/delivery/UPC_RO/OPENSER/BACKUP/RAW|NOT EXIST
Comptel_6_UAT|RO_OPEN_TAIFUN_DISTRIBUTOR|NodeParameters|Username|NOT EXIST|u1elkpe
Comptel_6_UAT|RO_OPEN_TAIFUN_DISTRIBUTOR|NodeParameters|Password|NOT EXIST|u1elkpe
Comptel_6_DEV2|AuditOutType|StreamParameterValue|Value|OPENSER_AUDIT|RO_AUDIT
Comptel_6_DEV2|EventLinkUserHome|StreamParameterValue|Value|NOT EXIST|/u01/appl/d2elkpe
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|Packages|EL_LIB_SFTP|NOT EXIST|1.2.1.
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|Packages|EL_CFG_FTP_DISTRIBUTOR|NOT EXIST|2.1.0.upc2
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|Packages|EL_APP_FTP_DISTRIBUTOR|NOT EXIST|2.1.5.
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|Username|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TransferType|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TransferRetryInterval|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TransferRetryCount|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TransferProtocol|NOT EXIST|Local
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TransferMode|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TempSuffix|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TempDirectory|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|TargetDirectory|NOT EXIST|${EventLinkUserHome}/delivery/UPC_PE_Rejected_Loader_stream/INPUT/RO
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|Ruleset|NOT EXIST|default_ruleset.pm
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|RemoteSuffix|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|RemotePrefix|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|RemoteOS|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|PublicKey|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|PrivateKey|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|Port|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|Password|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|MaximumSustainedNetworkDowntime|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|MaximumReconnectionInterval|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|Kex|NOT EXIST|
Comptel_6_DEV2|RO_OPEN_AUDIT_REJECTED_DISTRIBUTOR|NodeParameters|ExternalCommand|NOT EXIST|

Yoda · March 8, 2013, 9:58am

Here is the modified code:

sort file | awk -F'|' '
  (P[1] == $1) && (P[2] == $2) {
        n = split ($0, A, /[|]/)
        printf OFS
        for ( i = 3; i <= n; i++)
                printf OFS A;
        printf RS
        next;
} (P[1] != $1) || (P[2] != $2) {
        n = split ($0, A, /[|]/)
        print A[1], A[2]
        printf OFS
        for ( i = 3; i <= n; i++)
                printf OFS A;
        printf RS
} {
        n = split ($0, P, /[|]/)
} '

rajangupta2387 · March 8, 2013, 10:25am

Hi,

Thanks it is working but still i am now facing issues w.r.t to "printf" statement.

Input:

Comptel_6_UAT|IXBS_ENCODER|Packages|UPC_CFG_NL_DMS100_IXBS_ENCODER|NOT EXIST|2.1.1.
Comptel_6_UAT|IXBS_ENCODER|Packages|EL_LIB_FNL|NOT EXIST|1.5.1.
Comptel_6_UAT|IXBS_ENCODER|Packages|EL_LIB_CML|NOT EXIST|1.5.4.
Comptel_6_UAT|IXBS_ENCODER|Packages|EL_APP_MULTI_FORMAT_ENCODER|NOT EXIST|1.5.9.
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|UniqueOutputFileNames|NOT EXIST|1
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|SeparationCondition|NOT EXIST|EL_IXBS_SEPARATION
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|SeparateSourceID|NOT EXIST|0
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|SeparateSequenceNumbersInOutput|NOT EXIST|1
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|SeparateSequenceNumbersInFileNames|NOT EXIST|0
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|ResetInterval|NOT EXIST|0
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|RemoveFileTemplates|NOT EXIST|1
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|RemoveFileTemplatesTimeout|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|RecordWaitingTimeLimit|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|RecordLimit|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|RealTimeDelivery|NOT EXIST|0
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|OutputFileNameCreationMask|NOT EXIST|IXBS&CREATIONTIME(%Y%m%d%H%M%S)&FIRST(EL_OUT_FILE_SEQ)&FSIZE(9)&COUNT(*,6).CDR.&SOURCEID
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|OutputDescriptionFileName|NOT EXIST|UPC_NL_DMS100_IXBS_Description.xml
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|MinAgeOfFile|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|MaxAgeOfFile|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|InputFileNameExtractionMask|NOT EXIST|([a-zA-Z0-9]{7}_)(AMAUTH_0000)([0-9]{2})(_)([0-9]{14})
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|FileSizeLimit|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|CreateEmptyOutputFiles|NOT EXIST|0
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|CreateEmptyOutputFileOnSeparation|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|CreateEmptyOutputFileOnFlush|NOT EXIST|
Comptel_6_UAT|IXBS_ENCODER|NodeParameters|CloseFilesAtSchedule|NOT EXIST|1
Comptel_6_UAT|IXBS_ENCODER|Links|IXBS_ENCODER_OUT|NOT EXIST|1

I checked that printf is considering it to be operators. Do we have a solution for this as well?

Yoda · March 8, 2013, 10:36am

Yes, modify the printf statement, put format specifier:

printf "%s", OFS A;

rajangupta2387 · March 8, 2013, 10:56am

Thank you very much it worked.

pravin27 · March 11, 2013, 2:32am

How about this

 
sort File | awk -F"|" '{ if ( a!=$1 && b != $2 ) {print $1 OFS $2;} a=$1;b=$2; $1=$2=" " } 1' OFS=" "