Dot sourcing differences in ksh, AIX vs Linux vs Solaris

Why does dot sourcing of ksh functions behave so differently between AIX, Solaris, and Linux? How can I make Linux behave the way I want in the test I show below?

I have a library of interdependent functions I have developed and use in ksh in AIX. They also run in Solaris. Now I am migrating them to Linux. They aren't working properly in Linux. I have devised a test to illustrate the differences. I am looking for an explanation of the behavior, particularly in Linux, and how I can resolve it to be like Solaris (which would be OK) or like AIX (which would be ideal).

Test 1 below is the behavior I like. Test 2 is OK. Test 3 is the one I don't understand and need to fix.

I start by creating this file that contains the "MyFun" function definition in a directory that is on my PATH:

$ cat MyFun
#! /bin/ksh
print -- "In file MyFun, before def of function MyFun. PID=$$, system=$( uname -rsv )"
function MyFun
{
print -- "In function MyFun. PID=$$, system=$( uname -rsv )"
return $(( $# % 128 )) #number of arguments mod 128
}
print -- "In file MyFun, after def of function MyFun. PID=$$, system=$( uname -rsv )"

The contents above are stored in file "MyFun", which does NOT require execute permissions:

$ ls -l MyFun | xargs echo | cut -d ' ' -f 1,9 
-rw-r--r-- MyFun

############
TEST 1: run in AIX
(first install MyFun from above into a directory in your PATH, then cd to that directory)

Here when the file is dot sourced a second time, NONE of it executes. The system is caching the dot-sourcing operation, so if this function is shared by several other functions that dot source it and all are used in a script, it only executes once.

The important thing is that the inner part of the function never executes when the file is dot sourced (the function itself doesn't get called during dot sourcing). In this case the behavior is like an import (not include) statement, and is the behavior I want:

$ #dot source the file first time
$ . MyFun
In file MyFun, before def of function MyFun. PID=8908866, system=AIX 3 5
In file MyFun, after def of function MyFun. PID=8908866, system=AIX 3 5
$ #call the function after dot sourcing
$ MyFun
In function MyFun. PID=8908866, system=AIX 3 5
$ #dot source the file a second time
$ . MyFun
#(there is no output from the command above).

############
TEST 2: run in Solaris (first install MyFun from above)
(first install MyFun from above into a directory in your PATH, then cd to that directory)

Here when the file is dot sourced a second time, the file executes again just like the first time. The system is re-executing the entire file each time it is dot sourced, so if this function is shared by several other functions that dot source it and all are used in a script, the file re-executes every time and the function itself is redundantly redefined each time.

The important thing is that the inner part of the function still never executes when the file is dot sourced (the function itself doesn't get called during dot sourcing). This behaves like an include (not import) statement, and is inefficient compared to test 1, but still works:

$ #dot source the file first time
$ . MyFun
In file MyFun, before def of function MyFun. PID=29760, system=SunOS 5.10 Generic_141444-09
In file MyFun, after def of function MyFun. PID=29760, system=SunOS 5.10 Generic_141444-09
$ #call the function after dot sourcing
$ MyFun
In function MyFun. PID=29760, system=SunOS 5.10 Generic_141444-09
$ #dot source the file a second time
$ . MyFun
In file MyFun, before def of function MyFun. PID=29760, system=SunOS 5.10 Generic_141444-09
In file MyFun, after def of function MyFun. PID=29760, system=SunOS 5.10 Generic_141444-09

############
TEST 3: run in Linux (first install MyFun from above into a directory in your PATH, then cd to that directory)

Here when the file is dot sourced a second time, the FUNCTION BODY executes even though I don't want to CALL it.

The problem here is that the inner part of the function executes when the file is dot sourced after the first time (the function itself gets called during subsequent dot sourcing operations). I do not understand this behavior and it is harmful:

$ #dot source the file first time
$ . MyFun
In file MyFun, before def of function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
In file MyFun, after def of function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
$ #call the function after dot sourcing
$ MyFun
In function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
$ #dot source the file a second time
$ . MyFun
In function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011

I think you're using a non-portable function declaration... In some shells that would be a syntax error. This syntax works in any Bourne shell that supports functions AFAIK:

function () {
        contents
}
1 Like

I want the behavior given by using the function keyword as it is indeed different from that using the parentheses. I use the difference (ksh functions defined with function have their own function environment and my code relies heavily on that isolation, particularly for logging the name of the function, $0, being called), and I am not really interested in bash compatibility.

However I did notice that if I use corrected bash syntax I get basic compatibility with the Solaris test (regardless of whether the curly brace is on the same line as the parens):

$ cat MyFun
#! /bin/ksh
print -- "In file MyFun, before def of function MyFun. PID=$$, system=$( uname -rsv )"
MyFun() {
print -- "In function MyFun. PID=$$, system=$( uname -rsv )"
return $(( $# % 128 )) #number of arguments mod 128
}
print -- "In file MyFun, after def of function MyFun. PID=$$, system=$( uname -rsv )"

Running the above in Linux:

$ . MyFun
In file MyFun, before def of function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
In file MyFun, after def of function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
$ MyFun
In function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
$ . MyFun
In file MyFun, before def of function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
In file MyFun, after def of function MyFun. PID=37604, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011

The above still isn't acceptable because using the bash sytax, the function's name is not stored in $0 inside the function body when it executes.

Are you sure your Linux' /bin/ksh is really a ksh and not a link to somewhere else?

For my linux host:

 
$ which ksh
/bin/ksh

$ print ${.sh.version}
Version JM 93t+ 2010-06-21

ksh supports two types of function definition syntax:

 name() {body;}

result in a Posix style function. No scoping allowed and thus side-effects abound.

function name {body;}

Standard ksh functions. You can define local variables and have local signals.

With newer version ksh (such as inhabit linux) the . command has been expanded. In addition to sourcing a filename, the argument to the . command can now be a function. This invokes the function forcing it into local scope. This new feature is killing you. One idea is to name the file and the function it contains differently. . MyFunc.def would define a function called MyFunc. Then you have no name collision. Another idea is to use a pathname: . ./MyFunc .

The behavior you describe on AIX is strange. I can't explain that. But I think I could induce it. Try this:

if [[ $MyFunDef != "true" ]] ; then
        export MyFunDef="true"
        #! /bin/ksh  this line serves no purpose in a sourced file
        print -- "In file MyFun, before def of function MyFun. PID=$$, system=$( uname -rsv )"
        function MyFun
        {
        print -- "In function MyFun. PID=$$, system=$( uname -rsv )"
        return $(( $# % 128 )) #number of arguments mod 128
        }
        print -- "In file MyFun, after def of function MyFun. PID=$$, system=$( uname -rsv )"
fi
3 Likes

Well now this is promising. I will look into that in more detail overnight. Simply using the dot - dot-slash notation also makes the linux version at least compatible with Solaris. I modified the function slightly to show the value of $0 inside it here:

 
$ cat MyFun
#! /bin/ksh
print -- "In file MyFun, before def of function MyFun. PID=$$, system=$( uname -rsv )"
function MyFun {
  print -- "In function MyFun. \$0=$0. PID=$$, system=$( uname -rsv )"
  return $(( $# % 128 )) #number of arguments mod 128
}
print -- "In file MyFun, after def of function MyFun. PID=$$, system=$( uname -rsv )"

Running in linux:

$ . ./MyFun
In file MyFun, before def of function MyFun. PID=19620, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
In file MyFun, after def of function MyFun. PID=19620, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
$ MyFun
In function MyFun. $0=MyFun. PID=19620, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
$ . ./MyFun
In file MyFun, before def of function MyFun. PID=19620, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011
In file MyFun, after def of function MyFun. PID=19620, system=Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011

I now have a theory regarding the three behaviors that Charles reported. The ability to source a function was intended to be used with ksh style functions. These normally create their own scope and by sourcing them they behave like like posix style function.

Charles is defining a posix style function and then is sourcing that. This was not intended to happen. My theory is that AIX is advanced enough to support an early implementation of the function sourcing code. And in this implementation the attempt to source a posix style function is simply ignored.

And my theory continues that users were surprised that some functions could be sourced while other could not. And so, in the spirit of least surprise, ksh was modified to tolerate the sourcing of posix style functions.

The documenation does say that sourcing ksh style function is the feature that was added to ksh. The docs neither allow or prohibit sourcing a posix style function as far as I can see. Whether or not my theory is correct, sourcing a posix style function is an undocumented feature and Charles clearly has seen two different behaviors. I think it would be wise to stick only to the functionality that is explicitly allowed by the documentation.

Note - some versions of Linux had/have /usr/dash which is then symlinked to via /bin/ksh.
dash is NOT ksh. Do you have ksh88 or ksh93 on AIX? ksh88 is what you get as ksh on Solaris 9, 10, 11, by default. FWIW. I cannot be sure at all. But your Linux version appears not to be ksh93. dash was an early thing for Linux to get past the fact that ksh93 was not open source. Cygwin still has it in place of ksh.

Please post the output of (on Linux only):

ls -l /bin/ksh 
strings /bin/ksh | grep -i version

I don't know enough about AIX to comment on Perderabo's idea - it appears to have merit. IBM tends to be like Apple: everything on the command line has to do be redone/recoded/re-engineered, and then made to comply with POSIX later on.

PS: posting your flavor of Linux might have helped earlier on. Otherwise this was a very good question. Thanks.

"ksh" in AIX is a ksh88 which is unchanged since i work with AIX (v 3.2.3, ~1992), which is pretty long. To get a ksh93 you have to call "ksh93" explicitly.

One question i have for charles, though: why is it necessary to source anything? Why don't you use the "FPATH" variable and put your functions into a common path, similarly to a shared library?

FPATH works like the PATH, but for KornShell functions. put all your functions into separate files named the same as the function: if FPATH is set to "/foo/bar" the file "/foo/bar/boom" will contain the sole function "boom()". I use this for all my scripts in conjunction with a variable "DEVELOP", which switches between a copy of the library in my HOME and the common library. This way i can modify the lib functions without interrupting the production code:

#! /usr/bin/ksh

if [ -z "$DEVELOP" ] ; then
     . /usr/local/lib/ksh/StdEnv
else
     . ~/lib/StdEnv
fi

.... rest of the script

file /usr/local/lib/ksh/StdEnv contains:

if [ -z $DEVELOP ] ; then
     FPATH=/usr/local/lib/ksh
else
     FPATH=~/lib/ksh
fi

FOO=bar
...            # and all sorts of other environment settings

In my personal profile "$DEVELOP" is set so that scripts called with my user always use the local copy of the lib (or i unset it to use the common copy), all scripts started with non-personal users have "$DEVELOP" not set and therefore use the common library.

I hope this helps.

bakunin

1 Like