awk call in bash function called with arugments not working, something lost in translation?

Hello,

I have this awk code in a bash script to perform a find and replace task. This finds one unique line in a file and substitutes the found line with a replacement.

#! /bin/bash

# value determined elsewhere
total_outputs_p1=100
# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of replacement line
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"
# temporarily add variable values to environment varialbes
export comment_ch  look_for  replace_with

# process file with awk to find $look_for and replace with $replace_with
# if the first character(s) are are a comment, print the unmodified line
# if the line contains the string we are looking for, print the replacement line
# in all other cases, print the unmodified line
awk ' { if( substr($0,1,1) == ENVIRON["comment_ch"] || if( substr($0,1,2) == ENVIRON["comment_ch"] )
           print $0;
        else if($0 ~ ENVIRON["look_for"])
           print ENVIRON["replace_with"];
        else
           print $0;
      } ' $filename > tmp
# overwrite original file
mv tmp $filename

# clear variables set for this call to awk
unset ENVIRON["comment_ch"]
unset ENVIRON["look_for"]
unset ENVIRON["replace_with"]

This works well. It is rather involved for a simple find and replace but there are other conditions to consider. First, the line with $look_for is only unique in live code. The string may also exist in lines that have been commented out. Lines with a leading comment character need to be ignored. Also, $look_for often contains spaces and/or special characters that need to be escaped. I have had allot of trouble defining such values in awk using -v. Exporting the value of the string I am looking for to an environment variable allows me to use test the value in the awk call, including any spaces and necessary escape characters. I unset the environment variables immediately afterwords.

I have a dozen or so of these tasks to perform, so the logical structure is to have this code in a function to which the values for $filename , comment_ch , $look_for , and $replace_with are passed in the call. I have set it up in a function and printed the values from inside the function.

the code looks like,

#! /bin/bash

# function to call awk and implement find and replace
function find_and_replace () {

   # leading comment character (ignore line and print)
   local f_comment_ch=$1
   # line in file to find and replace
   local f_look_for=$2
   # replacement line
   local f_replace_with=$3
   # original filename
   local f_filename=$4

echo "     in function"
echo "  f_comment_ch: " $f_comment_ch
echo "    f_look_for: " $f_look_for
echo "f_replace_with: " $f_replace_with
echo "    f_filename: " $f_filename
echo ""

   # add variables to ENVIRON
   export f_comment_ch  f_look_for  f_replace_with

   # process file with awk to find $look_for and replace with $replace_with
   awk ' { if( substr($0,1,1) == ENVIRON["comment_ch"] || if( substr($0,1,2) == ENVIRON["comment_ch"] )
              print $0;
           else if($0 ~ ENVIRON["f_look_for"])
              print ENVIRON["f_replace_with"];
           else
              print $0;
         } ' $f_filename > tmp
   # overwrite original file
   mv tmp $f_filename

   # clear variables set for this call to awk
   unset ENVIRON["f_comment_ch"]
   unset ENVIRON["f_look_for"]
   unset ENVIRON["f_replace_with"]

}

# value determined elsewhere
total_outputs_p1=100
# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of replacement line
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"

# call find and replace
find_and_replace  "$comment_ch"  "$look_for"  "$replace_with"  "$filename"

This correctly prints the argument values from inside the function but the output I get is just a few lines with a single space. This doesn't make any sense. I am guessing that there may be some issue with the local environment inside the subshell for the function or the call to awk, but if ENVIRON["f_look_for"] evaluates to uninitialized, based on the logic in the awk code I would expect the entire unmodified file to be printed since neither the $comment_ch or $look_for would ever be found.

I'm not sure what to try next so I thought I would post.

Thanks,

LMHmedchem

I see a couple of potential issues with this approach.
I still don't understand why you can't use the usual paradigm of passing vars with -v to awk and have to use this somewhat convoluted approach.
Any chance you can attach a sample file?

I will put together a sample directory and post it.

The -v method works for the comments and replace line, no matter if there are special characters or spaces. It is the $look_for line that is the problem. This version of the code does not work. The $look_for line is never found.

#! /bin/bash

# value determined elsewhere
total_outputs_p1=100
# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of replacement line
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"

# process file with awk to find $look_for and replace with $replace_with
awk -v comment="$comment_ch" \
    -v find_line="$look_for" \
    -v replace_line="$replace_with" \
' { if( substr($0,1,1) == comment || if( substr($0,1,2) == comment )
       print $0;
    else if($0 ~ find_line)
       print replace_line;
    else
       print $0;
  } ' $filename > tmp
# overwrite original file
mv tmp $filename

If you replace,

else if($0 ~ find_line)

with,

else if($0 ~ / PARAMETER \(SIZEXX = /)

then it will function. This is hard coding specifically for each item to find and so can't be used in a function. I read that the -v option presented difficulties when using characters that need to be escaped and that is when I switched to the temporary environment variables that I read about on a stackexchange post (I think). I don't understand why it works, but not in the function.

I will post some samples in a few minutes. I need another cup of coffee.

LMHmedchem

for one I can see different number of blank spaces in look_for=" PARAMETER \(SIZEXX = " and in the inline version: else if($0 ~ / PARAMETER \(SIZEXX = /) ...

1 Like

There is a syntax error in the awk code:

if( substr($0,1,1) == comment || if( substr($0,1,2) == comment )

should probably be

if( substr($0,1,1) == comment || substr($0,1,2) == comment )
1 Like

This is probably just a copying error by me. I also fixed the error noted by Scrutinizer. I added the || because some comments are two characters.

I have attached a .zip with test files. The attachment has a directory with the script set_size_defs.sh and a sub-directory with the two files that are modified in this version. There is also a second sub-directory with a coppies of the files being modified as they are changed when the script is run.

This is working now, though I am not entirely sure what I fixed.

This is the working version of the script,

#! /bin/bash

# script to find lines and replace, lines commented out are ignored

# function to call awk and implement find and replace
function find_and_replace () {

   # comment character, lines to ignore and print with a leading comment
   local f_comment_ch=$1
   # line in file to find and replace
   local f_look_for=$2
   # replacement line
   local f_replace_with=$3
   # original filename
   local f_filename=$4

   # make sure the file exists
   if [ ! -f "$f_filename" ]; then
      echo ""
      echo $f_filename
      echo "cannot be found"
      echo ""
      exit 1
   fi

   # test print
   echo "   in function"
   echo "  f_comment_ch: " $f_comment_ch
   echo "    f_look_for: " $f_look_for
   echo "f_replace_with: " $f_replace_with
   echo "    f_filename: " $f_filename
   echo ""

   # add variables to ENVIRON
   export f_comment_ch  f_look_for  f_replace_with

   # process each file with awk to find and replace
   awk ' { if( substr($0,1,1) == ENVIRON["f_comment_ch"] ||
               substr($0,1,2) == ENVIRON["f_comment_ch"] )
              print $0;
           else if($0 ~ ENVIRON["f_look_for"])
              print ENVIRON["f_replace_with"];
           else
              print $0;
         } ' $f_filename > tmp
   # overwrite original file
   mv tmp $f_filename

   # clear variables set for this call to awk
   unset f_comment_ch
   unset f_look_for
   unset f_replace_with

}


#----------------------------------------------------------------------------------------
# this value is calculated by other code
total_outputs='2099'
# add 1 to total outputs for the rest of the values
total_outputs_p1=$(($total_outputs + 1))

#----------------------------------------------------------------------------------------
# replace value of SIZEXX in src_common_depend/UTILPARAMS.DAT

# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of line being replaced
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"

# call find and replace
find_and_replace  "$comment_ch"  "$look_for"  "$replace_with"  "$filename"


#----------------------------------------------------------------------------------------
# replace value of LASTINDEX in ./common_depend/sizedefs.h

# file being modified
filename='./common_depend/sizedefs.h'
# comment lines to ignore
comment_ch='//'
# line being searched for to replace, may contain escape characters
look_for="\#define LASTINDEX"
# value of line being replaced, characters do not need to be escaped here
replace_with="#define LASTINDEX   $total_outputs"

# call find and replace
find_and_replace  "$comment_ch"  "$look_for"  "$replace_with"  "$filename"

#----------------------------------------------------------------------------------------
echo "finished find and replace"
echo ""

This works and is quite fast. It will accommodate comments that are either 1 or 2 characters but the comments have to be at the beginning of the line unless I add code to trim leading whitespace. It doesn't matter if the replacement line has spaces or escaped characters.

I could go with this unless someone suggests the method is a bad idea.

LMHmedchem

1 Like

@LMHmedchem used to post a number of questions back at the old forums; but so far has not posted in the new forums.

:frowning:

1 Like