How to use same variable value inside as well as outside of the awk command?

Hi Jim,

The following script is in working state. But i m having one more problem with awk cmd. Could you tell me how to use any variable inside awk or how to take any variable value outside awk.
My problem is i want to maintain one property file in which i am declaring variable value into that property file and want to use that variable into awk script.
example.
file name:validate.property
INT_EUR=14
INT_TOK=54

now these two variable i want to use it into my shell script. new.sh which i wrote below.
Also one more thing .How to find scane arount 50000 records with minimum time. Actually the following script scan 42000 records in 6 min, i need to scan in 1 min is there any method in shell script.
i Hope u understand my problem. Could you please help me out.

Waiting for you reply.

#!/bin/bash
DATA_DIR=/export/opt/rtrupld/autosys/scripts_old
data=`ls $DATA_DIR/count.txt`


  awk ' END {print NR}' $data
  awk ' {          sum=0;

       print "Total lines",NR

      for(i=1; i<= length($0); i++) {if (substr($0,i,1)=="~" ) {sum++}  }
        if (NR !=1)
      {
       print "Total lines:::",NR
         print "oldsum :::",oldsum
         print "sum :::",sum
        if (oldsum==sum){
        print "oldsum",oldsum
      }
        else
        {
           if(oldsum!=sum)
              {print "error in ", $data," at line   ",NR,  " tilde
count=",sum}
            exit;
        } } else{
      oldsum=sum
      print "oldsum at last",oldsum
      }

      }'  $data

Please provide an example of the data that is in the $DATA_DIR/count.txt data file.

Why the need to invoke awk twice? Print the number of records at the top of the output?

Why do you need so many print statements? Why not just output the lines which are regarded as "errors"? Each print statement adds to the execution time.

Hi fpmurphy ,
Thanks for your response,

count.txt file contains following data,

INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~

Above record each line contains 21 ~ chars in line number 4th number o ~'s are 22 so my script should give line number 4 has an 22 ~'s also my script should take a static value from property file like validate.property
contains records like

INT_EUR=21
INT_TOK=21
INT_CA=54

so on
so that INT_EUR has value =21 so script will check is row 1 have ~=21 or not and so on every line should get scanned and does not match gives n error .
Could you please solve this problem.

Now your questions regarding why multiple awk commands. i am not be able to take any variable inside the awk command.Please guide me what to do.

Thanks & Regards,
Ganesh

---------- Post updated at 06:28 PM ---------- Previous update was at 12:11 AM ----------

Thanks,
And sorry for that. but i was placed queary again in details as someon asked me regading the problem in details.
Thanks for giving me this worning.

---------- Post updated 10-05-10 at 10:42 AM ---------- Previous update was 10-04-10 at 06:28 PM ----------

Hi ,
Could you please reply me on my thread.Its really helpfuul for me.

Thanks & Regards,
Ganesh

Something like this,

awk -v v1=`grep "INT_EUR" validate.property | cut -d"=" -f2` -F'~' '{if (NF-1 != v1) {print "line number "NR " has " NF-1 " ~"}}' count.txt

Hi Pravin,
Could you check my script i put it into forum. according to your solution i used above the my awk command. but it wont work. If you are online could you please send me that hardcodede values how to use inside in my awk command.

Thank & Regards
Ganesh

If you run the below command from your command prompt,

awk -v v1=`grep "INT_EUR" validate.property | cut -d"=" -f2` -F'~' '{if (NF-1 != v1) {print "Error in " $0 " at line number "NR " tilde count= " NF-1 }}' count.txt

You will get the o/p as below.
o/p

Error in INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~~ at line number 4 tilde count= 22

Is it your expected o/p, If no then post your expected o/p.

Assuming the value of the first line of your property file is determinative:

awk -F~ '
BEGIN{getline p < "property_file"; split(p,a,"=")} 
NF-1 != a[2] {print "Line " NR " has " NF " fields."}' file

Sorry All to disturb you again.
May be i m checking wrong. i need somthing like that.

I am using propety file as HandofValidation.property file which contains following data like

--The propety file has some constant which denotes that for EUR region 57 ~ count now from this the script should check whether in count.txt in row number 1 for EUR region ~ count are 57 are not if yes then check next row and so on.....

--Actually now my script gives proper result like in which line has more tilda than first row and gives an error line also .. But my boss want if first line has wrong ~ then my script will fail. So he want to keep all the constants in property file so that that count for perticular region can be take from property file.
--

--Below i am giving how i am using all the files you can give your idea on that.

--HandoffValidation.property
INTELLECT_EUR_DEAL_FLD_COUNT=57
INTELLECT_EUR_CASHFLOW_FLD_COUNT=14
INTELLECT_TOK_DEAL_FLD_COUNT=57
INTELLECT_TOK_CASHFLOW_FLD_COUNT=14
INTELLECT_NY_DEAL_FLD_COUNT=51
INTELLECT_NY_CASHFLOW_FLD_COUNT=14
INTELLECT_CA_DEAL_FLD_COUNT=51
INTELLECT_CA_CASHFLOW_FLD_COUNT=14

-count.txt
OPICS~CA~EOD~888~2010-10-01 01:01:29~20100930~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OPICS~CA~EOD~888~2010-10-01 01:01:29~20100930~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OPICS~CA~EOD~888~2010-10-01 01:01:29~20100930~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--my script
-new.sh

#!/bin/bash
WORK_DIR=/export/opt/rtrupld/autosys/scripts_old
DATA_DIR=/cdirect/rtrupld/incoming/intellect
#LOG=/export/opt/rtrupld/autosys/logs/impIntEurCshFlw.log
data=`ls $WORK_DIR/count.txt`
propfile=$WORK_DIR/HandoffValidation.properties

  awk ' END {print NR}' $data
  
    awk ' {          sum=0; 
  
       print"::INT:::::",var
 print "Total lines",NR
 for(i=1; i<= length($0); i++) {if (substr($0,i,1)=="~" ) {sum++} }  
        if (NR !=1)
 {
  print "Total lines:::",NR
         print "oldsum :::",oldsum
         print "sum :::",sum
        if (oldsum==sum){
        print "oldsum",oldsum
 }
        else
        {
           if(oldsum!=sum) 
              {print "error in ", $data," at line   ",NR,  " tilde count=",sum}
       exit;
        } } else{
 oldsum=sum
 print "oldsum at last",oldsum
 }
   
  }'  $data

--OPUTPUT
3
::INT:::::
Total lines 1
oldsum at last 52
::INT:::::
Total lines 2
Total lines::: 2
oldsum ::: 52
sum ::: 52
oldsum 52
::INT:::::
Total lines 3
Total lines::: 3
oldsum ::: 52
sum ::: 59
error in OPICS~CA~EOD~888~2010-10-01 01:01:29~20100930~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ at line 3 tilde count= 59

I need output like

above output but it should check variable value.
please suggest on my property file also. whether it is wrong or correct.
--Waiting for your reply.

Thanks & Regards
Ganesh

---------- Post updated at 02:34 PM ---------- Previous update was at 02:15 PM ----------

Hi pravin,
i am gettin following error after using your solution

grep: can't open .properties
awk: syntax error near line 1
awk: bailing out near line 1

--im using like

#!/bin/bash
WORK_DIR=/export/opt/rtrupld/autosys/scripts_old
DATA_DIR=/cdirect/rtrupld/incoming/intellect
#LOG=/export/opt/rtrupld/autosys/logs/impIntEurCshFlw.log
data=`ls $WORK_DIR/count.txt`
propfile=$WORK_DIR/HandoffValidation.properties
 
awk -v v1=`grep "INTELLECT_EUR_DEAL_FLD_COUNT"  $propfile | cut -d"=" -f2` -F'~' '{if (NF-1 != v1) {print "Error in " $0 " at line number "NR " tilde count= " NF-1 }}' $data

---------- Post updated at 03:56 PM ---------- Previous update was at 02:34 PM ----------

[/COLOR]Hi Franklin,
this is not double post of my thread.The reply which i got according to that i replied on that comments.I thought My problem has not getting cleared so i posted in detail what i actually need. in the last thread i explained everything in detail so anybady can understand the proble and give a exact solution.Excuse me if i broke the ruls .Please keep my thread open so i can get correct solution.
Thanks & regards,
Ganesh

HandoffValidation.property OR HandoffValidation.properties

is the below file file exist ?

/export/opt/rtrupld/autosys/scripts_old/HandoffValidation.properties

If yes then try this,

#!/bin/sh
WORK_DIR=/export/opt/rtrupld/autosys/scripts_old
DATA_DIR=/cdirect/rtrupld/incoming/intellect
#LOG=/export/opt/rtrupld/autosys/logs/impIntEurCshFlw.log
data=$WORK_DIR/count.txt
propfile=$WORK_DIR/HandoffValidation.properties

awk -v v1=`grep "INTELLECT_EUR_DEAL_FLD_COUNT" $propfile | cut -d"=" -f2` -F'~' '{if (NF-1 != v1) {print "Error in " $0 " at line number "NR " tilde count= " NF-1 }}' $data

Hi Pravin,
Its giving an error like

awk: syntax error near line 1
awk: bailing out near line 1

And what about my awk script no need to use.Also is this script scan 50 thousand records in 1 or 2 mins.

I used all the file name properly also correctd according to your comments and then tested.Giving error.Could you please check this.

Thanks & Regards,
Ganesh

Hi, Its working fine at my end. See the below code and o/p. So post the command which your typing

[root@powerbroker ~]# echo $data
count.txt
[root@powerbroker ~]# echo $propfile
HandoffValidation.properties
[root@powerbroker ~]# cat HandoffValidation.properties
INTELLECT_EUR_DEAL_FLD_COUNT=21
INTELLECT_EUR_CASHFLOW_FLD_COUNT=14
INTELLECT_TOK_DEAL_FLD_COUNT=57
INTELLECT_TOK_CASHFLOW_FLD_COUNT=14
INTELLECT_NY_DEAL_FLD_COUNT=51
INTELLECT_NY_CASHFLOW_FLD_COUNT=14
INTELLECT_CA_DEAL_FLD_COUNT=51
INTELLECT_CA_CASHFLOW_FLD_COUNT=14
[root@powerbroker ~]# cat count.txt
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~
INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~

Invocation

[root@powerbroker ~]# awk -v v1=`grep "INTELLECT_EUR_DEAL_FLD_COUNT" $propfile | cut -d"=" -f2` -F'~' '{if (NF-1 != v1) {print "Error in " $0 " at line number "NR " tilde count= " NF-1 }}' $data

O/P

Error in INTELLECT~JP~111~0~A~2jp10212991~RR~~~~~~~~~~~~~~~~ at line number 4 tilde count= 22

Hi Pravin,
Thanks for quick reply,
yes its working if i use "nawk" insted of "awk" but i have one more problem , As you see in properties file around 8 constants are there.
example.
INTELLECT_EUR_DEAL_FLD_COUNT=51
INTELLECT_EUR_CASHFLOW_FLD_COUNT=14
INTELLECT_TOK_DEAL_FLD_COUNT=57
INTELLECT_TOK_CASHFLOW_FLD_COUNT=14
INTELLECT_NY_DEAL_FLD_COUNT=51
INTELLECT_NY_CASHFLOW_FLD_COUNT=14
INTELLECT_CA_DEAL_FLD_COUNT=51
INTELLECT_CA_CASHFLOW_FLD_COUNT=14

then how to use these multiple constants in grep command so that no need to chang variable every tine in script , can we use variable over ther?

Thanks & Regards,
Ganesh

You want to verify 8 constant against count.txt ?
then try this,

#!/bin/sh
WORK_DIR=/export/opt/rtrupld/autosys/scripts_old
DATA_DIR=/cdirect/rtrupld/incoming/intellect
#LOG=/export/opt/rtrupld/autosys/logs/impIntEurCshFlw.log
data=$WORK_DIR/count.txt
propfile=$WORK_DIR/HandoffValidation.properties

while read line
do
field_name=`echo $line | cut -d"=" -f1`
cnt=`echo $line | cut -d"=" -f2`
echo "----------------------------------------------------------------"
echo "$line"
echo "----------------------------------------------------------------"
awk -v v1=$cnt -F'~' '{if (NF-1 != v1) {print "Error in " $0 " at line number "NR " tilde count= " NF-1 }}' $data
done < $propfile