Variable as input to awk command

Shell Programming and Scripting
1

Hi Gurus,

I need a suggestion, please help. I have a input file as below :
abc.txt :

*
xxxx:              00000
xxxxx:              00000
xxxx:              RANDOM
xxx:              RANDOM
**************************xxxxxxx***
*        abc
******************************
abc:
abc:              6213000
abx:            89234010
abc:             01179
******************************
*        acbxyz
******************************
Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2
0117943621,89234010001179436212,,621300020985821,,2347064000500,,,
0117943622,89234010001179436220,,621300020985822,,2347062347000,,,

I am using a perl script as below :

     1  #!usr/bin/perl -w
     2
     3  $test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' awktext.txt`;
     4  print $test2;
     5  $test2 = `awk 'BEGIN { count=0;}  { if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/) count++; else print "Unmatching line" ; print } END { print "Number of Lines = ",count;}' `;
     6  print $test2;

The above input file (abc.txt) that I have mentioned is an example file (since the original file will have upto 5 million records after the line : Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2)

Initially we had an perl script where we were validating each line by loading in array and running foreach for every line, after the below line in the input file (abc.txt): Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2
with an appropriate Regex as shown in the above perl script and we really faced performance issues. So, I was advised to use awk (though I see that it actually uses a new shell) for increase in performance, please suggest if you think otherwise. I have to use a perl script for few rules in organization.

Now, please suggest in the above script, how do i use the variable $test as input for the awk command ( i.e. $test2 = `awk 'BEGIN { count=0;} { if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/) count++; else print "Unmatching line" ; print } END { print "Number of Lines = ",count;}' abc.txt`;) since I want to process (run regex) the file for the regex only after the line as below (and not before it) : Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2

Running the awk as above will even process the above line and before it.

In short, how can I validate for Regex using a variable ($test1) as input to awk command and store the same in $test2.

If you have any other suggestion apart from the above, kindly let me know.

Thank you

2

I think your perl script should be like this (note the bold line).

#!usr/bin/perl -w

$test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' awktext.txt`;
print $test1;
$test2 = `awk 'BEGIN { count=0;}  { if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/) count++; else print "Unmatching line" ; print } END { print "Number of Lines = ",count;}' `;
print $test2;

Using the above code you've provided, you are using awk twice to parse the same data and that is the reason being slow. Instead, you could just use one awk command and parse the data and count the number of lines starting from a match of the following line

Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2

I think this is something you want to achieve? If that's the case, then you could either use awk or perl to accomplish.

awk --re-interval 'BEGIN { count=0; found=0 } { if(/Kitnb\/ICCID1\/ICCID2/) { found=1; next } if(/^([0-9]*\,){8}([0-9]*)$/ && found) { count++; } else if(! /^([0-9]*\,){8}([0-9]*)$/ && found) { print "Unmatching line"; print } } END { print "number of lines = " count } ' data.txt

To answer your original question and use the original code given, you could change it this way

#!usr/bin/perl -w

$test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' awktext.txt`;
print $test1;
$test2 = ` echo "$test1" | awk 'BEGIN { count=0;}  {  if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/)  count++; else print "Unmatching line" ; print } END { print "Number of  Lines = ",count;}' `;
print $test2;
3

For the below Suggestion I got the error as below.

awk --re-interval 'BEGIN { count=0; found=0 } { if(/Kitnb\/ICCID1\/ICCID2/) { found=1; next } if(/^([0-9]*\,){8}([0-9]*)$/ && found) { count++; } else
 if(! /^([0-9]*\,){8}([0-9]*)$/ && found) { print "Unmatching line"; print } } END { print "number of lines = " count } ' data.txt

=> perl Regex.pl

 Usage: awk [-F fs][-v Assignment][-f Progfile|Program][Assignment|File] ...

So, I modified the script as below (implementing the suggestion of using the variable) and it worked fine:

$test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' 10k.txt | awk 'BEGIN { count=0;} { if('"$regex"') count++; else print "Correct the output file, The line is :" ; print ;} END {} '`;

Thank you MR.bean :slight_smile:

Thanks,
Arun

---------- Post updated at 04:14 PM ---------- Previous update was at 04:13 PM ----------

BTW, could you please suggest why I got below error :

awk --re-interval 'BEGIN { count=0; found=0 } { if(/Kitnb\/ICCID1\/ICCID2/) { found=1; next } if(/^([0-9]*\,){8}([0-9]*)$/ && found) { count++; } else
 if(! /^([0-9]*\,){8}([0-9]*)$/ && found) { print "Unmatching line"; print } } END { print "number of lines = " count } ' data.txt

=> perl Regex.pl

 Usage: awk [-F fs][-v Assignment][-f Progfile|Program][Assignment|File] ...

Thanks,
Arun

4

It may be because your awk doesnt support --re-interval option.
Can you post the output of

awk --version

If you are on Solaris, try to run it with 'nawk'.