awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a
converted text file (original is a pdf).

1. Since the first two lines are (begin with) text they are removed
2. if $1 is a number then all text is merged (combined) into one line until the next number in $1. There might be no lines until the next number, or 1 line, 2 lines, etc. The amount of lines is variable but what is constant is the number in $1.
3. Since the last 3 lines are (begin with) text they are removed

I added a awk script attempt with description as well. Thank you :).

file

TIER 1 MOLECULAR PATHOLOGY PROCEDURES
The following codes represent gene-specific and genomic procedures.
81161 This code is out of order. See page 714.
81162 This code is out of order. See page 712.
81170 ABL1 (ABL proto-oncogene 1, non-receptor tyrosine kinase) (eg, acquired imatinib tyrosine kinase inhibitor
resistance), gene analysis, variants in the kinase domain
81200
ASPA (aspartoacylase) (eg, Canavan disease) gene analysis, common variants (eg, E285A, Y231X)
81201 APC (adenomatous polyposis coli) (eg, familial adenomatosis polyposis [FAP], attenuated FAP) gene
analysis; full gene sequence
81202 known familiar variants
81203 duplication/deletion variants
81205 BCKDHB (branched-chain keto acid dehydrogenase E1, beta polypeptide) (eg, Maple syrup urine disease)
gene analysis, common variants (eg, R183P, G278S, E422X)
81206 BCR/ABL1 (t(9;22)) (eg, chronic myelogenous leukemia) translocation analysis; major breakpoint,
qualitative or quantitative
CPT codes and descriptions only �2016 American Medical Association. All rights reserved.
CCI Comp. Code
Non-specific Procedure

desired output

81161 This code is out of order. See page 714.
81162 This code is out of order. See page 712.
81170 ABL1 (ABL proto-oncogene 1, non-receptor tyrosine kinase) (eg, acquired imatinib tyrosine kinase inhibitorresistance), gene analysis, variants in the kinase domain
81200 ASPA (aspartoacylase) (eg, Canavan disease) gene analysis, common variants (eg, E285A, Y231X)
81201 APC (adenomatous p
olyposis coli) (eg, familial adenomatosis polyposis [FAP], attenuated FAP) gene analysis; full gene sequence
81202 known familiar variants
81203 duplication/deletion variants
81205 BCKDHB (branched-chain keto acid dehydrogenase E1, beta polypeptide) (eg, Maple syrup urine disease) gene analysis, common variants (eg, R183P, G278S, E422X)
81206 BCR/ABL1 (t(9;22)) (eg, chronic myelogenous leukemia) translocation analysis; major breakpoint, qualitative or quantitative

awk

awk '$0==($0+0) {                        # remove lines that do not start with a number
       if ( $1 ~ /^[0-9]$/ )             # if $1 is a number
         if(l){print l;l=$0} {           # print line   
         else{l=l" "$0}}END{print l}     # if $1 is not a number combine line(l) until next number and print
                             }
                }' file                 

Hello cmccabe,

Could you please try following and let me know if this helps.

awk '/^[0-9]/{val=$0;getline;if($0 !~ /^[0-9]/){print val,$0} else {print val ORS $0}}'   Input_file

Thanks,
R. Singh

1 Like

One way:

awk 'NR>5{print A[NR%3]} {A[NR%3]=$0}' file |
awk '$1!~/[^0-9]/{if(p) print p; p=$0; next} {p=p OFS $0} END{print p}'

--
@Ravinder, that will fail in the following cases:

  • a broken line sequence that starts on an even line number.
  • The second part of a broken line begins with a number
  • The 3rd but last line is not broken, and has an even line number.
2 Likes

Thank you both very much.... @Scrutinizer would you mind adding a brief description of how the awk works. Thank you :).

---------- Post updated at 07:51 AM ---------- Previous update was at 07:23 AM ----------

Can the number in $1 be restrited to 5 digits? That is if there is a random number that is 3 digits in the start of the line it is removed.

all whitespace and symbols in $1 are removed
line 3 is removed because the random digit is less than a length of 5 digits

file

      81262        direct probe methodology (eg, Southern blot)
      81263    IGH@ (Immunoglobulin heavy chain locus) (eg, leukemia and lymphoma, B-cell), variable region somatic
               mutation analysis
714       l   New Code     s Revised Code       +   Add-On Code         Modifier -51 Exempt                  H    Telemedicine
                                                              CPT codes and descriptions only �2016 American Medical Association. All rights reserved.
                                                                                                            PATHOLOGY/ LABORATORY

desired output

81262        direct probe methodology (eg, Southern blot)
81263    IGH@ (Immunoglobulin heavy chain locus) (eg, leukemia and lymphoma, B-cell), variable region somatic
               mutation analysis

awk for leghth maybe:

awk 'NR>5{print A[NR%3]} {A[NR%3]=$0}' file | awk '{if(length($1) < 5 ) && $1!~/[^0-9]/ && { gsub(/[^[:alnum:]]/, "", $1);{if(p) print p; p=$0; next} {p=p OFS $0} END{print p}'

Thank you :).