awk - Why can't value of awk variables be passed to external functions ?

sreyan32 · November 22, 2014, 7:03am

I wrote a very simple script to understand how to call user-defined functions from within awk after reading this post.

function my_func_local {
echo "In func $1"
}

export -f my_func_local

echo $1 | awk -F"/" '{for (k=1;k<=NF;k++) {
if ($k == "a" ) {
system("my_local_func $k")
}
else{
print $k
}
}                        
}

I am running with the arguments-:
sh script.sh a/b/c/d/e
The expected output should be-:

In func a
b
c
d
e

But what I am getting is-:

In func
b
c
d
e

The value of k is not being passed into the external function when I am calling it from awk. Why is this happening ? How do you rectify this ? I need the function to be called within awk because what I am trying to do is take the input of a path from the user and recursively create the directories as I go.

How do I convert the value from $k to its actual value and pass it as a parameter ?

sea · November 22, 2014, 8:05am

As i see it, not really understanding awk yet, but you're comparing a STRING (a) with an INTEGER (k).
Anyhow, so you pass a list of variables, and when it matches a certain string, you want to call that function?

Must it be awk, because bash works just fine

#!/bin/bash
my_func_local() {
    echo "In func: my_func_local $1"
}

for ARG in "${@}";do
    [[ $ARG = a ]] && \
        my_func_local $ARG || \
        echo $ARG
done

sh script.sh a b c d
In func: my_func_local a
b
c
d

Hope this helps

RudiC · November 22, 2014, 8:05am

What you are getting surprises me when I look into the script you post:
If you'd define the function you call correctly, this might work better (my_func_local vs. my_local_func). And, don't put $k into the quotes to have it evaluated for the system call. And, awk uses sh to run the system call. Having symlinked /bin/sh to /bin/bash, I get

In func a
b
c
d
e

Corona688 · November 22, 2014, 9:57am

The answer is simple: awk is not shell. Variables don't work that way. $ means column here, and isn't even expanded inside quotes.

system("my_local_func " k)

sreyan32 · November 22, 2014, 1:00pm

rudic:

What you are getting surprises me when I look into the script you post:
If you'd define the function you call correctly, this might work better (my_func_local vs. my_local_func). And, don't put $k into the quotes to have it evaluated for the system call. And, awk uses sh to run the system call. Having symlinked /bin/sh to /bin/bash, I get
In func a
b
c
d
e

Can you please specify in details what you have done to get the result ? can you post your entire script which shows the symlink ?

---------- Post updated at 11:26 PM ---------- Previous update was at 11:18 PM ----------

What you have told me to do doesn't work. I am using utilizing the following code now-:

function my_local_function {
echo $1
}

export -f my_local_function

echo $1 | awk -F"/" '{ for (k=1;k<=NF;k++)
{
if ( $k == "a" ) {
    system("my_local_function" k)
        }
else {
    print $k
    }
    }
}'

I am getting the following erroneous error-:

sh: my_local_function1: command not found
b
c
d

---------- Post updated at 11:30 PM ---------- Previous update was at 11:26 PM ----------

sea:

As i see it, not really understanding awk yet, but you're comparing a STRING (a) with an INTEGER (k).
Anyhow, so you pass a list of variables, and when it matches a certain string, you want to call that function?

Must it be awk, because bash works just fine
#!/bin/bash
my_func_local() {
   echo "In func: my_func_local $1"
}

for ARG in "${@}";do
   [[ $ARG = a ]] && \
   my_func_local $ARG || \
   echo $ARG
done
sh script.sh a b c d
In func: my_func_local a
b
c
d
Hope this helps

Can you please explain to me what your script means step by step ? Sorry for being blunt but I am comfortable with awk rather than raw shell scripting. However for this problem it does not matter what I use so if you can explain how your script works then it would be fine for my purpose.

RudiC · November 22, 2014, 2:27pm

As requested:

sudo ln -sf /bin/bash /bin/sh
echo "a/b/c/d/e" |
awk -F"/"       '{for (k=1;k<=NF;k++)   {if ($k == "a" )        {system("my_func_local " $k)}
                                         else                   {print $k}
                                        }
                 }
                '
In func a
b
c
d
e

Don't forget to recreate the original sh symlink.

sreyan32 · November 22, 2014, 2:52pm

rudic:

As required:

sudo ln -sf /bin/bash /bin/sh
echo "a/b/c/d/e" |
awk -F"/"       '{for (k=1;k<=NF;k++)   {if ($k == "a" )        {system("my_func_local " $k)}
   else                   {print $k}
   }
   }
   '
In func a
b
c
d
e

Don't forget to recreate the original sh symlink.

Thanks. But can you tell me why did you need to create a symlink /bin/sh for this to work ? I mean whats the logic behind this. Why wasn't it working before ?

MadeInGermany · November 22, 2014, 10:17pm

'exported functions' is a (questionable) bash-only feature.
system() runs /bin/sh so it only works if /bin/sh is bash.
--
Changing /bin/sh is a risk - e,g. system boot scripts might fail!

sreyan32 · November 23, 2014, 4:39am

So what should I do ?

Corona688 · November 23, 2014, 1:50pm

You messed up te same way in many places. Remember awk is not shell -- $k means column k. Stop using $k when you don't want column. Start using k.

Corona688 · November 23, 2014, 1:51pm

Ideally? You'd stop doing that.

RudiC · November 24, 2014, 3:05pm

Using shell functions from within awk is difficult as you can see and it doesn't make too much sense. Write the same function in awk and use it in there!

sreyan32 · November 25, 2014, 6:19am

I am trying to get a concrete idea on what awk can and cannot do by pushing its less used and documented features to the limit.

Also, the symlink that you have provided in the beginning of the script using ln -sf . Is that permanent ? I mean will it persist even after script has ended ?

RudiC · November 25, 2014, 6:31am

Yes, that symlink is a permanent modification in the file system, nothing to do with the script. I only added it for demonstration purposes and told you to be careful and reset it after trying it.

BTW, you can't get back anything but the exit code from a system call inside awk.

Corona688 · November 25, 2014, 10:03am

This is not an awk feature, just a bash feature.

Also, you're not really "calling" bash functions. You're generating an entire new shell which calls the functions. This means you can't use the function to set shell variables or anything -- they will vanish when this new shell quits.

sreyan32 · November 29, 2014, 7:13am

Sir, I have tried all possible combinations of $k and k. And neither of them give me my expected output. Either I am not able grasp what you are telling me or you don't understand what I want. Or maybe it can't be done all together.

I posting various combinations of scripts that I have tried with their respective outputs-:

First Sample Script-: (Where all $k has been replaced with k )

 
 function my_local_function () {
 echo $1;
 }
 export -f my_local_function
 echo $1 | awk -F"/" '{ for (k=1;k<=NF;k++) { if ( k == "a" ) { system("my_local_function" k) } else { print k } } }'

Output-:

As you can see above no errors but I don't get my expected output. Since I am using k and not $k . Since k represents the loop variable and not its value.

Second Sample Script-: (Where I am using system("my_local_function" k) and $k in all other places)

 
 function my_local_function () {
 echo $1;
 }
 export -f my_local_function
 echo $1 | awk -F"/" '{ for (k=1;k<=NF;k++) { if ( $k == "a" ) { system("my_local_function" k) } else { print $k } } }'

Output-:

sh: my_local_function1: command not found
b
c
d

Third Sample script: (Where I am using system("my_local_function k") and $k in all other places)

 
 function my_local_function () {
 echo $1;
 }
 export -f my_local_function
 echo $1 | awk -F"/" '{ for (k=1;k<=NF;k++) { if ( $k == "a" ) { system("my_local_function k") } else { print $k } } }'

Output-:

k
b
c
d

Again not a correct output but error free.

Fourth Sample Script-: (Where I am using system("my_local_function $k") and $k in all other places)

 
 function my_local_function () {
 echo $1;
 }
 export -f my_local_function
 echo $1 | awk -F"/" '{ for (k=1;k<=NF;k++) { if ( $k == "a" ) { system("my_local_function $k") } else { print $k } } }'

Output-:


b
c
d

One blank line and the rest of the output is OK.

So basically what I want to do is call a BASH function from awk and pass the value of an awk variable as an argument. CAN IT BE DONE ?

And please can you post an entire working script next time rather than telling me what to do. It will avoid any further confusions.

derekludwig · November 29, 2014, 7:35am

Your "sample two script" had two typos in the systemcall, a space before the trailing quote of "my_local_function" and a missing '$' before the 'k':

function my_local_function () { echo @ $1; }
export -f my_local_function
echo $1 | awk -F"/" '{ for (k=1;k<=NF;k++) { if ($k == "a") { system("my_local_function " $k) } else { print $k } } }'

I added a '@' to the local function just for clarity:

$ bash x a/b/c/d/e
@ a
b
c
d
e

Is this what was desired?

Please note that you'll be creating a subshell everytime you want to invoke the local function. An alternative to consider:

function my_local_function () { echo @ $1; }
export -f my_local_function
echo $1 \
| awk -F"/" '{ for (k=1;k<=NF;k++) print ( $k == "a" ? "my_local_function" : "echo" ), $k; }' \
| while read line; do
    eval $line
done

sea · November 29, 2014, 7:41am

sreyan32:

---------- Post updated at 11:30 PM ---------- Previous update was at 11:26 PM ----------
sea;302926246:
As i see it, not really understanding awk yet, but you're comparing a STRING (a) with an INTEGER (k).
Anyhow, so you pass a list of variables, and when it matches a certain string, you want to call that function?

Must it be awk, because bash works just fine
#!/bin/bash
my_func_local() {
   echo "In func: my_func_local $1"
}

for ARG in "${@}";do
   [[ $ARG = a ]] && \
   my_func_local $ARG || \
   echo $ARG
done
sh script.sh a b c d
In func: my_func_local a
b
c
d
Hope this helps
Can you please explain to me what your script means step by step ? Sorry for being blunt but I am comfortable with awk rather than raw shell scripting. However for this problem it does not matter what I use so if you can explain how your script works then it would be fine for my purpose.

Sure:

#!/bin/bash
# Definition of a function, as you already done
my_func_local() {
    echo "In func: my_func_local $1"
}

# Regular parsing through all passed arguments
# This reads one passed argument after another into the variable ARG
for ARG in "${@}";do
    # Compare current ARG-ument with letter 'a', 
    #   IF SO (&&), call 'my_func_local a', 
    #   otherwise (||) just print current ARG
    [[ $ARG = a ]] && \
        my_func_local $ARG || \
        echo $ARG
done

Hope this helps

derekludwig · November 29, 2014, 7:50am

Sea,

The for loop need to be:

IFS="${IFS}/"
for ARG in ${@}; do

so the arguments can be separated by the field separator "/".

MadeInGermany · December 1, 2014, 4:30pm

And, because of unquoted "$@", it is safer to do

set -f # disable globbing on wildcards

and nevertheless quote "$ARG", and use if-then-else because && - || is harder to read and a non-zero exit code in the && branch will continue with the || branch.

#!/bin/bash
# Definition of a function, as you already done
my_func_local() {
    echo "In func: my_func_local $1"
}

# Regular parsing through all passed arguments
# This reads one passed argument after another into the variable ARG

# By putting the following in a ( subshell ) we limit the scope of the next two statements to the subshell
(
set -f # disable globbing on wildcards
IFS="${IFS}/"
for ARG in $@ ;do
    # Compare current ARG-ument with letter a
    if [[ "$ARG" = "a" ]]; then
        my_func_local "$ARG"
    else
        echo "$ARG"
    fi
done
)
# subshell ended, IFS and settings restored.