Getline not working in awk

Shell Programming and Scripting
1

hi,
i am trying to parse a file in awk to generate a output to be written in a file depeding upon some condition. below is the code

Content of file1

919873741577,9131638459976206,20130715150109,S,919811000214,2A65,405899136999995
91225,9132019667696305,20130715150110,S,USSDLIKE,161A,405899136999995
54321,9132008640045585,20130715150109,S,SYMBIOTIC,2470,405899136999995
54321,9132008640041216,20130715150108,S,SYMBIOTIC,2458,405899136999995
54321,9132008640045246,20130715150109,S,SYMBIOTIC,2470,405899136999995
919865754417,9194*665665,20130715161714,F,2374,,919875089998
918926048523,911401727253,20130715141326,F,405899153999998,,919875089998
918640868045,911401727253,20130715165419,F,405899153999999,,919875089998

Code for parsing is as follows

 awk -F, -v p2pcdr="file1" -v p2preject="P2P_REJ.txt"
                        ' BEGIN{
                                while ((getline line < p2pcdr ) > 0 )
                                {
                                        if ( $2 ~ "[*,#]" || ( ($2 + 0) !~ "^[0,5,7,8,9]")  || length($2 + 0) <4 || ($2 !~ "^[0,5,7,8,9]"))
                                                P2PREJECTCDR[line]
                                        else if ( (length($6) == 13 || length($6) == 14 || length($6)==11) && !($6 ~ "^[A-Z,a-z]") )
                                                P2PREJECTCDR[line]
                                        else if ($6 =="" && substr($2,1,4)=="0091" && substr($2 + 0,3,1) !~ "^[1,2,5,7,8,9]")
                                                P2PREJECTCDR[line]
                                        else if ( $6 == "" && substr($2,1,2) == "00" && length($2) == "12" )
                                                P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]
                                        else if ( $6 == "" && substr($2,1,1) == "0" && length($2) == "11" )
                                                P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]
                                        else if ( $6 == "" && substr($2,1,3) == "+91" && length($2) == "13" )
                                                P2PCDR[$1,$2,$3,$4,$5,substr($2,2,12),$7]
                                        else if ( $6 == "" && substr($2,1,2) == "91" && length($2) == "12" && (substr($2,3,1) ~ "^[5,7,8,9]") )
                                                P2PCDR[$1,$2,$3,$4,$5,$2,$7]
                                        else if ($5 != "" && $6 != "")
                                                P2PCDR[line]
                                        else if ($5 == "" && $6 != "")
                                                P2PCDR[line]
                                        else if ($6 == "" && length($2)<=10 && length($2 + 0) >= 4 )
                                                P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]
                                        else
                                                P2PREJECTCDR[line]
                                } close (p2pcdr)
                             }   
                                 END
                                {
                                        for ( counter in P2PCDR )
                                                print counter >> "P2P.txt"
                                        for ( counter in P2PREJECTCDR )
                                                print counter >> p2preject
                                }
                        '

can you please help as this is not working

2

There is a fundamental mis-understanding how awk works.
Your code reads an entire file in the BEGIN section,
and then skips the main section with the built-in loop over the file(s) given in the awk arguments.

3

yes, there is :confused: .
But can you please suggest how to get this thing rectified.

4

Do you know the joke "could you heal this cow", showing the doctor a steak?
Please give more information!
What is the role of the 2nd file?
What is the expected outcome?

5

if you could see the input file, first 5 lines will be valid as per the conditions mentioned in the begin block & that is the reason it is being redirected to P2P.txt in the END block, whereas last 3 lines of the input file will be redirected to the invalid data & this invalid data will be pushed to other file "P2P_REJ.txt"

inputfile file1(is comma separated)

919873741577,9131638459976206,20130715150109,S,919811000214,2A65,405899136999995
91225,9132019667696305,20130715150110,S,USSDLIKE,161A,405899136999995
54321,9132008640045585,20130715150109,S,SYMBIOTIC,2470,405899136999995
54321,9132008640041216,20130715150108,S,SYMBIOTIC,2458,405899136999995
54321,9132008640045246,20130715150109,S,SYMBIOTIC,2470,405899136999995
919865754417,9194*665665,20130715161714,F,2374,,919875089998
91892604852

3,911401727253,20130715141326,F,405899153999998,,919875089998
918640868045,911401727253,20130715165419,F,405899153999999,,919875089998
[/CODE]

1st Output required based on conditions in P2P.txt

919873741577,9131638459976206,20130715150109,S,919811000214,2A65,405899136999995
91225,9132019667696305,20130715150110,S,USSDLIKE,161A,405899136999995
54321,9132008640045585,20130715150109,S,SYMBIOTIC,2470,405899136999995
54321,9132008640041216,20130715150108,S,SYMBIOTIC,2458,405899136999995
54321,9132008640045246,20130715150109,S,SYMBIOTIC,2470,405899136999995

2nd Output required based on conditions in P2P_REJ.txt would contain 3 lines based from input file

919865754417,9194*665665,20130715161714,F,2374,,919875089998
918926048523,911401727253,20130715141326,F,405899153999998,,919875089998
918640868045,911401727253,20130715165419,F,405899153999999,,919875089998

writing the actual code again for reference

awk -F "," -v p2pcdr="file1" -v p2preject="P2P_REJ.txt" '
BEGIN {
		while ((getline line<p2pcdr ) > 0 )
		{
			if ( $2 ~ "[*,#]" || ( ($2 + 0) !~ "^[0,5,7,8,9]")  || length($2 + 0) <4 || ($2 !~ "^[0,5,7,8,9]"))
                                                P2PREJECTCDR[line]
			else if ( (length($6) == 13 || length($6) == 14 || length($6)==11) && !($6 ~ "^[A-Z,a-z]") )
                                                P2PREJECTCDR[line]
			else if ($6 =="" && substr($2,1,4)=="0091" && substr($2 + 0,3,1) !~ "^[1,2,5,7,8,9]")
                                                P2PREJECTCDR[line]
                                       	else if ($6 =="" && substr($2,1,2)=="91" && length($2) !=12 && length($2) !=16 && substr($2,3,1) !~ "^[5]")
                                                P2PREJECTCDR[line]
                                        else if ( substr($2,1,2)=="91" && (length(substr($2,3,length($2)-2) ) < 4) )
                                                P2PREJECTCDR[line]
                                        else if ( substr($2,1,4)=="0091" && (length(substr($2+0,3,length($2)-2) ) < 4) )
                                                P2PREJECTCDR[line]
                                        else if ($6!="" && length($6)>10 && $6 ~ "^[1,2,3,4,6,7,8]")
                                                P2PREJECTCDR[line]
                                        else if ( $6!="" && length($6)!=10 && length($6)!=12 && length($6)!=4 && length($6)!=6 && $6!~"^[A-Z,a-z]" && substr($6,1,2)!="91" )
                                                P2PREJECTCDR[line]
                                        else if ($6!="" && length($6)==10 && $6 ~ "^[1,2,4,6]")
                                                P2PREJECTCDR[line]
                                        else if ( $6 == "" && substr($2,1,4) == "0091" )
                                                P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]
                                        else if ( $6 == "" && substr($2,1,2) == "91" && length($2) == "16" )
                                                P2PCDR[$1,$2,$3,$4,$5,substr($2,3,4),$7]
                                        else if ( $6 == "" && substr($2,1,2) == "00" && length($2) == "12" )
                                                P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]
                                        else if ( $6 == "" && substr($2,1,1) == "0" && length($2) == "11" )
                                                P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]
                                        else if ( $6 == "" && substr($2,1,3) == "+91" && length($2) == "13" )
                                                P2PCDR[$1,$2,$3,$4,$5,substr($2,2,12),$7]
                                        else if ( $6 == "" && substr($2,1,2) == "91" && length($2) == "12" && (substr($2,3,1) ~ "^[5,7,8,9]") )
                                                P2PCDR[$1,$2,$3,$4,$5,$2,$7]
                                        else if ($5 != "" && $6 != "")
                                                P2PCDR[line]
                                        else if ($5 == "" && $6 != "")
                                                P2PCDR[line]
                                        else if ($6 == "" && length($2)<=10 && length($2 + 0) >= 4 )
                                                P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]
                                        else
                                                P2PREJECTCDR[line]
		} close (p2pcdr)
	 END
                                {
                                        for ( counter in P2PCDR )
                                                print counter >> "P2P.txt"
                                        for ( counter in P2PREJECTCDR )
                                                print counter >> p2preject
                                }
}'
6

Like I said, replace the BEGIN by a main loop over the first file argument (file1).
The variable line becomes $0, and $1,$2 etc. are the fields of $0.
Further, awk does not accept if ... else if ... without deeper { nesting }. (This is unlike shell and C.)
I have replaced with next that directly jumps to the next cycle (next line).
I further have replaced the outer { braces } that allows to use implicit if.
Also awk does not accept a new line everywhere. Here is my result:

awk -F, -v p2preject="P2P_REJ.txt" '
  $2 ~ "[*,#]" || ( ($2 + 0) !~ "^[0,5,7,8,9]")  || length($2 + 0) <4 || ($2 !~ "^[0,5,7,8,9]") { P2PREJECTCDR[$0]; next }
  (length($6) == 13 || length($6) == 14 || length($6)==11) && !($6 ~ "^[A-Z,a-z]") { P2PREJECTCDR[$0]; next }
  $6 =="" && substr($2,1,4)=="0091" && substr($2 + 0,3,1) !~ "^[1,2,5,7,8,9]" { P2PREJECTCDR[$0]; next }
  $6 == "" && substr($2,1,2) == "00" && length($2) == "12" { P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]; next }
  $6 == "" && substr($2,1,1) == "0" && length($2) == "11" { P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]; next }
  $6 == "" && substr($2,1,3) == "+91" && length($2) == "13" { P2PCDR[$1,$2,$3,$4,$5,substr($2,2,12),$7]; next }
  $6 == "" && substr($2,1,2) == "91" && length($2) == "12" && (substr($2,3,1) ~ "^[5,7,8,9]") { P2PCDR[$1,$2,$3,$4,$5,$2,$7]; next }
  $5 != "" && $6 != "" { P2PCDR[$0]; next }
  $5 == "" && $6 != "" { P2PCDR[$0]; next }
  $6 == "" && length($2)<=10 && length($2 + 0) >= 4 { P2PCDR[$1,$2,$3,$4,$5,$2 + 0,$7]; next }
  { P2PREJECTCDR[$0] }
  END {
    for ( counter in P2PCDR ) print counter >> "P2P.txt"
    for ( counter in P2PREJECTCDR ) print counter >> p2preject
  }
' file1
7

Thanks made in germany

8

Hi Madein germany,
The solution works perfectly fine, since the file has more than 30 million records, so can you suggest is there any way to improve the awk script for faster processing.

9

There are a couple of optimizations that I think could improve speed

  • One is that there is no need to use arrays to store values and print them in the end. If you need to filter double values you can pipe the output through sort -u ( or use arrays in extra if statements before the print statements..)
  • Two is that the logic could be simplified further I think which would help both speed and readability.

I tried to simplify a bit, I think it does more ore less what you want, at least with your input sample it renders the same output. I am sure it could be a reduced a bit further still. Have a look and adjust as needed...

awk -F, -v p2preject="P2P_REJ.txt" '
  ( $2 ~ /[*#]|^[12346]/ || length($2 + 0) < 4 ) ||
  ( length($6) ~ /^(11|13|14)$/ && $6 !~ /^[A-Za-z]/ ) ||
  ( $6 =="" && $2~/^0091[0346]/ ) {
    print > p2preject
    next
  }

  $6 != "" {
    print
    next
  }
  $2~/^(00|\+91|91[5789]|0)..........$/ || ( length($2)<=10  && length($2 + 0) >= 4 ) {
    print $1,$2,$3,$4,$5,$2 + 0,$7
    next
  }

  {
    print > p2preject
  }

' file > P2P.txt
10

Help me out: wouldn't a numeric compare $2 < 1000 be much faster than this strange construct length ($2+0) < 4 (which involves an addition, number - string conversion, char counting). What be the advantage of this construct (unless there'd be decimals)?

11

Hi RudiC, I did a couple further suggestions for optimization/simplification of the code suggested by MadeinGermany in post #6, which was an optimization of the OP's code to begin with. So I did not introduce length ($2+0) < 4 . And like I said it could be probably be optimized bit further still (like your suggestion surely is), but I drew the line somewhere...