awk to combine lines if fields match in lines

In the awk below, what I am attempting to do is check each line in the tab-delimeted input , which has ~20 lines in it, for a keyword
SVTYPE=Fusion . If the keyword is found I am splitting $3 using the . (dot) and reading the portion before and after the dot in an array a .
If it does have that keyword is in line 1 then the it will also be in the line directly below it will (always and in the same location/format). The same process is performed on that
line except the split is read into array b .
Then if array b matches array a the desired output is printed tab-delimeted .
If the line does not have the word in it then that line is skipped and appears as-is in the output (nothing needs to be done to it, but it is printed in the output). I hope the awk is close, I included comments as to what I think is happening in each line. The two lines in bold and the two lines in italics should produce the desired output, the ones in color do not match and can be skipped.Thank you :).

input

chr12    12006495    ETV6-NTRK3.E4N15.COSF823.1_1    G    G]chr15:88483984]    .    PASS    SVTYPE=Fusion;READ_COUNT=1868;GENE_NAME=ETV6;EXON_NUM=4;RPM=1.5825e-09;NORM_COUNT=0.001582480886121524;ANNOTATION=COSF823;FUNC=[{'gene':'ETV6','exon':'4','oncomineGeneClass':'Gain-of-Function','oncomineVariantClass':'Fusion'}]    GT:GQ    ./.:.
chr15    88483984    ETV6-NTRK3.E4N15.COSF823.1_2    T    ]chr12:12006495]T    .    PASS    SVTYPE=Fusion;READ_COUNT=1868;GENE_NAME=NTRK3;EXON_NUM=15;RPM=1.5825e-09;NORM_COUNT=0.001582480886121524;ANNOTATION=COSF823;FUNC=[{'gene':'NTRK3','exon':'15','oncomineGeneClass':'Gain-of-Function','oncomineVariantClass':'Fusion'}]    GT:GQ    ./.:.
chr12    12022903    ETV6-NTRK3.E5N15.COSF571.1_1    G    G]chr15:88483984]    .    PASS    SVTYPE=Fusion;READ_COUNT=414833;GENE_NAME=ETV6;EXON_NUM=5;RPM=3.5143e-07;NORM_COUNT=0.3514268166126607;ANNOTATION=COSF571;FUNC=[{'gene':'ETV6','exon':'5','oncomineGeneClass':'Gain-of-Function','oncomineVariantClass':'Fusion'}]    GT:GQ    ./.:.
chr15    88483984    ETV6-NTRK3.E5N15.COSF571.1_2    T    ]chr12:12022903]T    .    PASS    SVTYPE=Fusion;READ_COUNT=414833;GENE_NAME=NTRK3;EXON_NUM=15;RPM=3.5143e-07;NORM_COUNT=0.3514268166126607;ANNOTATION=COSF571;FUNC=[{'gene':'NTRK3','exon':'15','oncomineGeneClass':'Gain-of-Function','oncomineVariantClass':'Fusion'}]    GT:GQ    ./.:.
chr17    7577108    COSM10749;COSM43737    C    A,T    149.594    PASS    AF=0.0830415,0.0;AO=372,2;DP=4420;FAO=166,0;FDP=1999;FR=.,.,REALIGNEDx0.0865;FRO=1833;FSAF=82,0;FSAR=84,0;FSRF=952;FSRR=881;FWDB=0.0072184,-0.0207142;FXX=4.99998E-4;HRUN=1,1;LEN=1,1;MLLD=293.795,80.5366;OALT=A,T;OID=COSM10749,COSM43737;OMAPALT=A,T;OPOS=7577108,7577108;OREF=C,C;PB=.,.;PBP=.,.;QD=0.299338;RBI=0.00721997,0.02565;REFB=1.40155E-4,-7.81395E-4;REVB=1.50579E-4,0.0151276;RO=4043;SAF=187,1;SAR=185,1;SRF=2118;SRR=1925;SSEN=0,0;SSEP=0,0;SSSB=-0.0251826,-5.12306E-4;STB=0.52327,0.5;STBP=0.541,1.0;TYPE=snp,snp;VARB=-0.00153404,0.0;HS;FUNC=[{'origPos':'7577108','origRef':'C','normalizedRef':'C','gene':'TP53','normalizedPos':'7577108','normalizedAlt':'A','polyphen':'1.0','gt':'pos','codon':'TTT','coding':'c.830G>T','sift':'0.0','grantham':'205.0','transcript':'NM_000546.5','function':'missense','protein':'p.Cys277Phe','location':'exonic','origAlt':'A','exon':'8','oncomineGeneClass':'Loss-of-Function','oncomineVariantClass':'Hotspot'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:149:4420:1999:4043:1833:372,2:166,0:0.0830415,0.0:185,1:187,1:2118:1925:84,0:82,0:952:881:1
chr10    89624278    .    G    T    62.8836    PASS    AF=0.0785393;AO=297;DP=4155;FAO=157;FDP=1999;FR=.;FRO=1842;FSAF=77;FSAR=80;FSRF=908;FSRR=934;FWDB=0.0113997;FXX=4.99998E-4;HRUN=1;LEN=1;MLLD=117.237;OALT=T;OID=.;OMAPALT=T;OPOS=89624278;OREF=G;PB=.;PBP=.;QD=0.12583;RBI=0.040843;REFB=5.39678E-4;REVB=-0.0392199;RO=3844;SAF=150;SAR=147;SRF=1936;SRR=1908;SSEN=0;SSEP=0;SSSB=0.00159791;STB=0.502301;STBP=0.96;TYPE=snp;VARB=-0.00676678;FUNC=[{'origPos':'89624278','origRef':'G','normalizedRef':'G','gene':'PTEN','normalizedPos':'89624278','normalizedAlt':'T','gt':'pos','codon':'TAG','coding':'c.52G>T','transcript':'NM_000314.4','function':'nonsense','protein':'p.Glu18Ter','location':'exonic','origAlt':'T','exon':'1'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:62:4155:1999:3844:1842:297:157:0.0785393:147:150:1936:1908:80:77:908:934:1

awk with comments

awk -F'\t' -v OFS="\t"' # call awk and set FS and OFS as tab
{
match($0,/SVTYPE=[^;]*/); # look in line for SVTYPE
SVTYPE_VALUE=substr($0,RSTART+7,RLENGTH-7); # index SVTYPE for Fusion (value to look for)
if(SVTYPE_VALUE == "Fusion"){ # if SVTYPE is Fusion
{split($3,a,"."); a=a[1]"."a[2]}' # split the string in $3 at using the . and store the portion before the . and after (ETV6-NTRK3.E4N15) in array
a else if(SVTYPE_VALUE !~ "Fusion") # if SVTYPE not Fusion
next; # skip to next line
}
}
}
{
match($0,/SVTYPE=[^;]*/); # look in line for SVTYPE
SVTYPE_VALUE=substr($0,RSTART+7,RLENGTH-7); # index SVTYPE for Fusion (value to look for)
if(SVTYPE_VALUE == "Fusion"){ # if SVTYPE is Fusion
{split($3,b,"."); b=b[1]"."b[2]}' # split the string in $3 at using the . and store the portion before the . and after (ETV6-NTRK3.E4N15) in array
b else if(SVTYPE_VALUE !~ "Fusion") # if SVTYPE not Fusion
next; # skip to next line
} 
} 
}
} 
} 
}
{
for(j in b){ # define loop
for(i=1;i<=length(a);i++){ # iterate through loop
if(b[j] ~ a){ # compare array b to array a for match
print --- desired output ----
}
}
}' input | awk 'BEGIN {print "Position\tLocus\tClass\tFunction\tGene\tCount"}' > output # add header

desired output

Locus    Class    Function    Gene    Count
chr12:12006495-chr15:88483984    Fusion    Gain-of-function    ETV6E4-NTRK3E15    1868
chr12:12022903-chr15:88483984Fusion    Gain-of-function    ETV6E5-NTRK3E15    414833   
chr17    7577108    COSM10749;COSM43737    C    A,T    149.594    PASS    AF=0.0830415,0.0;AO=372,2;DP=4420;FAO=166,0;FDP=1999;FR=.,.,REALIGNEDx0.0865;FRO=1833;FSAF=82,0;FSAR=84,0;FSRF=952;FSRR=881;FWDB=0.0072184,-0.0207142;FXX=4.99998E-4;HRUN=1,1;LEN=1,1;MLLD=293.795,80.5366;OALT=A,T;OID=COSM10749,COSM43737;OMAPALT=A,T;OPOS=7577108,7577108;OREF=C,C;PB=.,.;PBP=.,.;QD=0.299338;RBI=0.00721997,0.02565;REFB=1.40155E-4,-7.81395E-4;REVB=1.50579E-4,0.0151276;RO=4043;SAF=187,1;SAR=185,1;SRF=2118;SRR=1925;SSEN=0,0;SSEP=0,0;SSSB=-0.0251826,-5.12306E-4;STB=0.52327,0.5;STBP=0.541,1.0;TYPE=snp,snp;VARB=-0.00153404,0.0;HS;FUNC=[{'origPos':'7577108','origRef':'C','normalizedRef':'C','gene':'TP53','normalizedPos':'7577108','normalizedAlt':'A','polyphen':'1.0','gt':'pos','codon':'TTT','coding':'c.830G>T','sift':'0.0','grantham':'205.0','transcript':'NM_000546.5','function':'missense','protein':'p.Cys277Phe','location':'exonic','origAlt':'A','exon':'8','oncomineGeneClass':'Loss-of-Function','oncomineVariantClass':'Hotspot'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:149:4420:1999:4043:1833:372,2:166,0:0.0830415,0.0:185,1:187,1:2118:1925:84,0:82,0:952:881:1
chr10    89624278    .    G    T    62.8836    PASS    AF=0.0785393;AO=297;DP=4155;FAO=157;FDP=1999;FR=.;FRO=1842;FSAF=77;FSAR=80;FSRF=908;FSRR=934;FWDB=0.0113997;FXX=4.99998E-4;HRUN=1;LEN=1;MLLD=117.237;OALT=T;OID=.;OMAPALT=T;OPOS=89624278;OREF=G;PB=.;PBP=.;QD=0.12583;RBI=0.040843;REFB=5.39678E-4;REVB=-0.0392199;RO=3844;SAF=150;SAR=147;SRF=1936;SRR=1908;SSEN=0;SSEP=0;SSSB=0.00159791;STB=0.502301;STBP=0.96;TYPE=snp;VARB=-0.00676678;FUNC=[{'origPos':'89624278','origRef':'G','normalizedRef':'G','gene':'PTEN','normalizedPos':'89624278','normalizedAlt':'T','gt':'pos','codon':'TAG','coding':'c.52G>T','transcript':'NM_000314.4','function':'nonsense','protein':'p.Glu18Ter','location':'exonic','origAlt':'T','exon':'1'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:62:4155:1999:3844:1842:297:157:0.0785393:147:150:1936:1908:80:77:908:934:1 

description of output

header line
$1:$2 of line 1 - $1:$2 of line 2     SVTYPE=(value)     oncomineGeneClass=(value)     array a and array b match (value)     READ_Count=(value)   
lines that were nor processed

Hello cmccabe,

Could you please check if following may help you in same.

awk '/SVTYPE=Fusion/{match($5,/].*]/);sub(/.COS.*/,"",$3);sub(/-/,"",$3);sub(/\./,"-",$3);VAL=substr($5,RSTART+1,RLENGTH-2);num=split($8, array,";");for(i=1;i<=num;i++){if(array ~  /SVTYPE/){sub(/.*=/,"",array);svtype=array};if(array ~ /READ_COUNT/){sub(/.*=/,"",array);read_count=array}};match($0,/oncomineGeneClass.*,/);print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;}'  Input_file

EDIT: Adding a non-one liner form of solution too now.

awk '/SVTYPE=Fusion/{
                        match($5,/].*]/);
                        sub(/.COS.*/,"",$3);
                        sub(/-/,"",$3);
                        sub(/\./,"-",$3);
                        VAL=substr($5,RSTART+1,RLENGTH-2);
                        num=split($8, array,";");
                        for(i=1;i<=num;i++){
                                                if(array ~  /SVTYPE/){
                                                sub(/.*=/,"",array);
                                                svtype=array
                                                                        };
                                                if(array ~ /READ_COUNT/){
                                                sub(/.*=/,"",array);
                                                read_count=array
                                                                           }
                                           };
                        match($0,/oncomineGeneClass.*,/);
                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;
                    }
    '   Input_file
 

Thanks,
R. Singh

1 Like

The output is very close, but only line 1 and 3 are printed because line 2 matches 1 and line 4 matches 3.

output of script

 chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr15:88483984-chr12:12006495 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr15:88483984-chr12:12022903 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
 

desired output

 chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
 chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
 chr17 entire line.... (this line does not have the keyword in it so is printed in the output)
 chr10 entire line...   (this line does not have the keyword in it so is printed in the output
 

Would you be able to comment each line so I may better understand? Thank you very much :).

awk '/SVTYPE=Fusion/{   # regex to look in line for SVTYPE=Fusion
                        match($5,/].*]/);  # grab the $5 value after the ] but before theouter ] (chr15:88483984)
                        sub(/.COS.*/,"",$3);  # split on the . before COS and grag the value in $3 (ETV6-NTRK3.E4N15)
                        sub(/-/,"",$3);   # split on the - and grab the text to the left (ETV6) 
                        sub(/\./,"-",$3);   # split on the - and grab the text to the right and add a hypen before (-NTRK3)
                        VAL=substr($5,RSTART+1,RLENGTH-2);    # not sure haven't got the concept of RSTART and RLENGTH
                        num=split($8, array,";");  # read the $8 value into array up to the ; (SVTYPE=Fusion}
                        for(i=1;i<=num;i++){  # create loop to iterate next line
                                                if(array ~  /SVTYPE/){ # check array 
                                                sub(/.*=/,"",array);    # matches in next line
                                                svtype=array  # define match criteria
                                                                        };
                                                if(array ~ /READ_COUNT/){   #  strore value of READ_COUNT in array 
                                                sub(/.*=/,"",array);   #  value of READ_COUNT in array (1868)
                                                read_count=array    # define match and print to line
                                                                           }
                                           }; 
                        match($0,/oncomineGeneClass.*,/);  # grab(Gain-of-function
                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;  # print desired output
                    }
    '   Input_file

I do not really understand RSTART and RLENGTH fully but believe they define the index of a field.

Hello cmccabe,

Could you please go through following and let me know if this helps you.

awk '/SVTYPE=Fusion/{                                                                                                 #### Searching the string SVTYPE=Fusion on each line. If it is found then perform following actions.
                        match($5,/].*]/);                                                                             #### Using match functionality by providing the regex where it will match from ] to till ] in 5th field.
                        sub(/.COS.*/,"",$3);                                                                          #### Substituting the .COS.* means from .COS to everything to NULL in 3rd field.
                        sub(/-/,"",$3);                                                                               #### Substituting -(dash) with NULL in 3rd field.
                        sub(/\./,"-",$3);                                                                             #### Substituting DOT with - in 3rd field.
                        VAL=substr($5,RSTART+1,RLENGTH-2);                                                            #### Putting the values of subtring whose value starts from RSTART+1 to RLENGTH-2, here point to be noted RSTART and RLENGTH are the OOTB awk's keywords which will be SET when a match is found, we have done this match for 5th field above.
                        num=split($8, array,";");                                                                     #### Splitting field 8th whose delimiter is ";" and storing it's length(array's length) to variable named num. 
                        for(i=1;i<=num;i++){                                                                          #### Starting a loop which will start from value 1 of variable i to till value of variable num.
                                                if(array ~  /SVTYPE/){                                             #### Checking here condition if any array's value is equal to /SVTYPE/ then do following.
                                                sub(/.*=/,"",array);                                               #### If above condition is TRUE then substituting the .*= everything till = to NULL in array's value.
                                                svtype=array;                                                      #### Keeping array's value in variable named svtype.
                                                                        };
                                                if(array ~ /READ_COUNT/){                                          #### Checking another condition where checking if array a's value is equal to /READ_COUNT/ then perform following actions.
                                                sub(/.*=/,"",array);                                               #### substituting everything till = to NULL in array's value.
                                                read_count=array                                                   #### Storing value of array's in read_count.
                                                                           }
                                           };
                        match($0,/oncomineGeneClass.*,/);                                                             #### Using match function of awk to perform a match from  /oncomineGeneClass.*/ here.
                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;   #### printing the values of $1, $2, VAL(which is having the values of 1st match which we performed above for $5), svtype, subtring of above used match for /oncomineGeneClass.* then read_count values.
                    }
    '   Input_file                                                                                                    #### Mentioning the Input_file.

NOTE: Kindly DO NOT run the above code it is only for explanation purposes.

Thanks,
R. Singh

1 Like

I have added two lines to add a header and remove duplicates based on $4 . The portion of code in bold below was an attempt to print the two lines (in this example, there may be more). If the SVTYPE=Fusion is not true then the entire line is printed in the output. Thank you :).

awk -v OFS="\t" '/SVTYPE=Fusion/{
                        match($5,/].*]/);
                        sub(/.COS.*/,"",$3);
                        sub(/-/,"",$3);
                        sub(/\./,"-",$3);
                        VAL=substr($5,RSTART+1,RLENGTH-2);
                        num=split($8, array,";");
                        for(i=1;i<=num;i++){
                                                if(array ~  /SVTYPE/){
                                                sub(/.*=/,"",array);
                                                svtype=array
                                                         
               };
                                                if(array ~ /READ_COUNT/){
                                                sub(/.*=/,"",array);
                                                read_count=array
                                                                           }
                                           };
                        match($0,/oncomineGeneClass.*,/);
                        print "Locus\tType\tFunction\tGene\tReads" ### add header
                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;
                        ' $8 != "\"SVTYPE=Fusion\"" {print;}'
                    }
' input | awk -F'\t' '!seen[$4]++'### remove duplicates based on $4

awk: cmd. line:21:                         
awk: cmd. line:21:                         ^ unexpected newline or end of string

current output

chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833

desired output

chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr17 entire line.... (this line does not have the keyword in it so is printed in the output)
chr10 entire line...   (this line does not have the keyword in it so is printed in the output

Hello cmccabe,

If you simply want to print the lines which doesn't have condition of string fusion then following may help you in same(I haven't tested it though).

awk '/SVTYPE=Fusion/{
                        match($5,/].*]/);
                        sub(/.COS.*/,"",$3);
                        sub(/-/,"",$3);
                        sub(/\./,"-",$3);
                        VAL=substr($5,RSTART+1,RLENGTH-2);
                        num=split($8, array,";");
                        for(i=1;i<=num;i++){
                                                if(array ~  /SVTYPE/){
                                                sub(/.*=/,"",array);
                                                svtype=array
                                                                        };
                                                if(array ~ /READ_COUNT/){
                                                sub(/.*=/,"",array);
                                                read_count=array
                                                                           }
                                           };
                        match($0,/oncomineGeneClass.*,/);
                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;
			next
                    }
    1
    '   Input_file

Thanks,
R. Singh

1 Like

The output seems to be only 3 of the 4 lines (in this example there are 4 lines, but that is variable depending on the input.
Is the next only processing one of the lines because they both do not have SVTYPE=Fusion so it only sees one line?. Thank you :).

Locus    Type    Function    Gene    Reads
chr12:12006495-chr15:88483984    Fusion    Gain-of-Function    ETV6NTRK3-E4N15    1868
chr12:12022903-chr15:88483984    Fusion    Gain-of-Function    ETV6NTRK3-E5N15    414833
chr17    7577108    COSM10749;COSM43737    C    A,T    149.594    PASS    AF=0.0830415,0.0;AO=372,2;DP=4420;FAO=166,0;FDP=1999;FR=.,.,REALIGNEDx0.0865;FRO=1833;FSAF=82,0;FSAR=84,0;FSRF=952;FSRR=881;FWDB=0.0072184,-0.0207142;FXX=4.99998E-4;HRUN=1,1;LEN=1,1;MLLD=293.795,80.5366;OALT=A,T;OID=COSM10749,COSM43737;OMAPALT=A,T;OPOS=7577108,7577108;OREF=C,C;PB=.,.;PBP=.,.;QD=0.299338;RBI=0.00721997,0.02565;REFB=1.40155E-4,-7.81395E-4;REVB=1.50579E-4,0.0151276;RO=4043;SAF=187,1;SAR=185,1;SRF=2118;SRR=1925;SSEN=0,0;SSEP=0,0;SSSB=-0.0251826,-5.12306E-4;STB=0.52327,0.5;STBP=0.541,1.0;TYPE=snp,snp;VARB=-0.00153404,0.0;HS;FUNC=[{'origPos':'7577108','origRef':'C','normalizedRef':'C','gene':'TP53','normalizedPos':'7577108','normalizedAlt':'A','polyphen':'1.0','gt':'pos','codon':'TTT','coding':'c.830G>T','sift':'0.0','grantham':'205.0','transcript':'NM_000546.5','function':'missense','protein':'p.Cys277Phe','location':'exonic','origAlt':'A','exon':'8','oncomineGeneClass':'Loss-of-Function','oncomineVariantClass':'Hotspot'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:149:4420:1999:4043:1833:372,2:166,0:0.0830415,0.0:185,1:187,1:2118:1925:84,0:82,0:952:881:1

Hello cmccabe,

When I run my script I got following output. Where 4 lines in which string Fusion is there is coming to output.

chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr15:88483984-chr12:12006495 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr15:88483984-chr12:12022903 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr17    7577108    COSM10749;COSM43737    C    A,T    149.594    PASS    AF=0.0830415,0.0;AO=372,2;DP=4420;FAO=166,0;FDP=1999;FR=.,.,REALIGNEDx0.0865;FRO=1833;FSAF=82,0;FSAR=84,0;FSRF=952;FSRR=881;FWDB=0.0072184,-0.0207142;FXX=4.99998E-4;HRUN=1,1;LEN=1,1;MLLD=293.795,80.5366;OALT=A,T;OID=COSM10749,COSM43737;OMAPALT=A,T;OPOS=7577108,7577108;OREF=C,C;PB=.,.;PBP=.,.;QD=0.299338;RBI=0.00721997,0.02565;REFB=1.40155E-4,-7.81395E-4;REVB=1.50579E-4,0.0151276;RO=4043;SAF=187,1;SAR=185,1;SRF=2118;SRR=1925;SSEN=0,0;SSEP=0,0;SSSB=-0.0251826,-5.12306E-4;STB=0.52327,0.5;STBP=0.541,1.0;TYPE=snp,snp;VARB=-0.00153404,0.0;HS;FUNC=[{'origPos':'7577108','origRef':'C','normalizedRef':'C','gene':'TP53','normalizedPos':'7577108','normalizedAlt':'A','polyphen':'1.0','gt':'pos','codon':'TTT','coding':'c.830G>T','sift':'0.0','grantham':'205.0','transcript':'NM_000546.5','function':'missense','protein':'p.Cys277Phe','location':'exonic','origAlt':'A','exon':'8','oncomineGeneClass':'Loss-of-Function','oncomineVariantClass':'Hotspot'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:149:4420:1999:4043:1833:372,2:166,0:0.0830415,0.0:185,1:187,1:2118:1925:84,0:82,0:952:881:1
chr10    89624278    .    G    T    62.8836    PASS    AF=0.0785393;AO=297;DP=4155;FAO=157;FDP=1999;FR=.;FRO=1842;FSAF=77;FSAR=80;FSRF=908;FSRR=934;FWDB=0.0113997;FXX=4.99998E-4;HRUN=1;LEN=1;MLLD=117.237;OALT=T;OID=.;OMAPALT=T;OPOS=89624278;OREF=G;PB=.;PBP=.;QD=0.12583;RBI=0.040843;REFB=5.39678E-4;REVB=-0.0392199;RO=3844;SAF=150;SAR=147;SRF=1936;SRR=1908;SSEN=0;SSEP=0;SSSB=0.00159791;STB=0.502301;STBP=0.96;TYPE=snp;VARB=-0.00676678;FUNC=[{'origPos':'89624278','origRef':'G','normalizedRef':'G','gene':'PTEN','normalizedPos':'89624278','normalizedAlt':'T','gt':'pos','codon':'TAG','coding':'c.52G>T','transcript':'NM_000314.4','function':'nonsense','protein':'p.Glu18Ter','location':'exonic','origAlt':'T','exon':'1'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:62:4155:1999:3844:1842:297:157:0.0785393:147:150:1936:1908:80:77:908:934:1

Thanks,
R. Singh

1 Like

Yes, I see now... it is the awk -F'\t' '!seen[$4]++' that is removing the 4th line.

chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr15:88483984-chr12:12006495 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr15:88483984-chr12:12022903 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833

Since the lines in bold have the same $4 value only one of the lines needs to be printed.
The lines in italics have the same. This will only apply to those lines where SVTYPE=Fusion .

would adding the code in purple only remove the duplicates in the lines where SVTYPE=Fusion ? Thank you very much :).

   };
                        match($0,/oncomineGeneClass.*,/);
                        print "Locus\tType\tFunction\tGene\tReads"
                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;
                                   '{ if (a[$4]++ == 0) print $0; }' "$@"
            next
                    }
    1

desired output

chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr17 entire line.... (this line does not have the keyword in it so is printed in the output)
chr10 entire line...   (this line does not have the keyword in it so is printed in the output

Hello cmccabe,

Based on your shown sample output, could you please try following and let me know if this helps you.

awk '/SVTYPE=Fusion/{
                        match($5,/].*]/);
                        sub(/.COS.*/,"",$3);
                        sub(/-/,"",$3);
                        sub(/\./,"-",$3);
                        VAL=substr($5,RSTART+1,RLENGTH-2);
                        num=split($8, array,";");
                        for(i=1;i<=num;i++){
                                                if(array ~  /SVTYPE/){
                                                sub(/.*=/,"",array);
                                                svtype=array
                                                                        };
                                                if(array ~ /READ_COUNT/){
                                                sub(/.*=/,"",array);
                                                read_count=array
                                                                           }
                                           };
                        match($0,/oncomineGeneClass.*,/);
                        if(++A[$3]==1)
                                    {
                                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;
                                    }
                    next
                    }
        1
    '    Input_file
 

Thanks,
R. Singh

1 Like

Thank you as always for the great help and explanations :).

---------- Post updated at 07:30 AM ---------- Previous update was at 07:00 AM ----------

I am still a bit confused on this concept:

I guess my confusion is what defines RSTART+1 and RLENGTH-2 in this line but RSTART+20 and RLENGH-22 in the print?

In the first RSTART $5 was used as the substring, so chr15:88483984 , so is RSTART+1 the chr15 and RLENGTH-2 = to :88483984 ? Thank you :).

Hello cmccabe,

No need to be confuse :slight_smile:

I will provide a guidance here and you could play with it. Let us assume you are printing simply RSTART and RLENGTH ( RSTART will keep the index of starting point of the REGEX and RLENGTH will have the last character etc's index in it) variable's values then you will see some additional characters/digits are coming(which are not in your requirement) so this + and - to them is the adjustment of moving the cursor to print the exact required output. I hope I was clear in it, if you have doubt then kindly do let me know on same, will try to explain more on same.

Thanks,
R. Singh

1 Like

Thank you very much that helps a lot :).