Count Pattern

Shell Programming and Scripting
1

I need to count the total no of SINY and SINN patterns from the below file , The file does not have a record terminators , data in the file is streamed i.e. one line

I want to put this count in xml structure

<count>
<value>$totalcount</value>
</count>

Tried the following awk statements but since the file does not have any record terminators the output of count is always coming out as 0

awk -F "" '/SIN\*Y\*/{x++;}END{print x >>"count.txt"}' Test.txt

 
count.txt |awk 'BEGIN{print"<Totalcount>"}{print "<value>"$1"</value>"}END { print "</Totalcount>"}'> reccount.txt

Input file Test.txt

 
ISA*xx*xyzzzz    *00*          *0x*xxxxxxxxx      *14*xxxxxxxxxxxxx  *140417*1005*^*00501*xxxxxxxxx*0*P*>~GS*xx*xxxxxxxxx*xxxxxxxxxxxx1*20140417*xxxxxx*xxx*X*005010X220A1~ST*xxx*xxxxxx*005010X220A1~BGN*00*xxxxxxxxxxxxxx*20140417*1005*Ex***4~N1*P5*xxxxxxxxxxxxx*FI*xxxxxxxxx~N1*IN*xxxxxxxx*FI*xxxxxxxxx~SIN*Y*18*xx*XN*A~REF*0F*xxxxxxxxx~REF*1L*xxxxxxxx~REF*ZZ*xxxxxxxxxxxxx~NM1*IL*1*xxxxx*xxxxxxxx*A***ZZ*00~PER*IP**HP*xxxxxxxxxx~N3*xxxxxxxxxxxxxxxxx~N4*xxxxxxxxxxxxxxx*208771193~DMG*xx*xxxxxxxx*x~HD*xxx**xxx*xxx~DTP*xxx*xx*xxxxxxxx~LX*1~NM1*xxxxxxxxx*XX*xxxxxxxxxx*72~PER*xxx*TE*xxxxxxxxxx*FX*xxxxxxxxxx~SIN*N*xx*xxx*XN*A~REF*xx*3xxxxxxxx~REF*xx*xxxxxxx2~REF*xxx00xxxxxxxxxxx~NM1*xxxx*Rxxxxxxx*xxxxxxxx*R***xx*0x~PER*xx**xx*xxxxxxxxxx~N3*xxxxxxxxxxxxxxxxxxD~N4*xxxxxxxxxx*xx*2xxxxxxx8~DMG*xx*xxxxxxx0*x~HD*xx1**xxO*xxx~DTP*xxx*xx*xxxxxxx2~LX*1~NM1*xxx2*xxxxxxxxx Mxxxxxxxxxxxxxxx-xx*******xx~PER*xx**xxxxxxxxxxxx7*xxx*xxxxxxxx6~SE*xx*xxxxxx~GE*x*xxx~IEA*x*0000xxxxx~

Thanks!

2

If your grep has -o, you can try

$ grep -o 'SIN\*[YN]' input | wc -l
2

otherwise you need to be more resourceful. perhaps:

$ sed 's/SIN\*[YN]/\n&\n/g' input | grep -cx 'SIN\*[YN]'
2
$ perl -lne 'END {print $c} $c += s/SIN\*[YN]//g' input
2
3

awk

$ awk '{s += gsub(/SIN\*[YN]/,"&")}END{print s}' file
2

$ awk '{s += gsub(/SIN\*[YN]/,"&")}END{f="count.txt";print s >f; close(f)}' test.txt
4

oh and keeping awk, perhaps:

Removed. Akshay beat me to it
5

Thanks Akshay!

Can you please explain the code

6 
awk '{s += gsub(/SIN\*[YN]/,"&")}END{print s}' file

function gsub returns the number of substitutions made, variable s holds sum of this count and printed in END block

7

What is & used for ?

 
awk '{s += gsub(/SIN\*[YN]/,"&")}END{print s}' fileawk '{s += gsub(/SIN\*[YN]/,"&")}END{print s}' file
8

In a substitution such as gsub in awk or s in sed, & means the matched text itself.

9 
 gsub(r, s [, t])        For each substring matching the regular expres-
                               sion r in the string t, substitute  the  string
                               s,  and return the number of substitutions.  If
                               t is  not  supplied,  use  $0.   An  &  in  the
                           replacement text is replaced with the text that
                           was actually matched.  Use \& to get a  literal
                               &.   (This  must  be  typed as "\\&"; see GAWK:
                               Effective AWK Programming for a fuller  discus-
                               sion  of  the  rules for &'s and backslashes in
                               the replacement text of sub(), gsub(), and gen-
                               sub().)
10

Thanks for the explanation!

How can I put the value of the count in the below xml structure

 
<count>
<value>$totalcount</value>
</count>

I tried this statement but the xml structure is looping for each input record

count.txt |awk 'BEGIN{print"<Totalcount>"}{print "<value>"$1"</value>"}END { print "</Totalcount>"}'> reccount.txt
11

why don't you combine all in one like this

awk '{s += gsub(/SIN\*[YN]/,"&")}END{if(s)print "<Totalcount><value>"s"</value></Totalcount>"}' file
12

Thank you Akshay and Neutronscott!

13

For 90MB file the above command runs for several minutes , is there a way we can reduce the processing time ?

Thanks

14

Can anyone optimize the above command , I am ruuning a 90 MB file and it is taking good amount of time to get the count of the pattern

Thanks