AIX UNIX (kshell) to Linux Shell Script Migration.

Raghuraman.R · June 17, 2014, 7:19am

Hi,

Could you please let me know what kind of changes/issues commonly occurs at Scripting /command level during AIX Unix (kshell) to Linux shell script migration.

bakunin · June 17, 2014, 7:37am

AIX has a ksh88 as system shell and a ksh93 in /usr/bin/ksh93 Korn Shell (the real ksh - not "pdksh" and similar clones with heavy shortcomings) is available as free software and you should be able to get a package suitable for your system (.deb or .rpm, most probably) and install it.

There should be no difference betwwen the Korn Shell on AIX and any other ksh.

I hope this helps.

bakunin

Raghuraman.R · June 17, 2014, 8:03am

Hi bakunin,

Thanks for your response. So you suggeste to install ksh88 , ksh93 on new Linux Server inorder to have existing Application Scripts to work fine.

bakunin · June 17, 2014, 9:14am

Basically: yes. There might probably be no ksh88 package any more, but because ksh93 is downwards compatible with ksh88 you can install ksh93 and everything should run fine.

Of course "run fine" only means that the shell syntax is fine. If you use a AIX-specific command (like "lsuser", for instance) or option (AIX grep has a "-p" option the standard grep lacks, for instance) somewhere you will get an error saying "command not found" or "illegal soption" or something such.

I hope this helps.

bakunin

Kibou · June 17, 2014, 9:43am

If you want to make your scripts work in bash, I am currently doing a migration from AIX to RHEL and we've found these issues:

Variables in AIX inside a test can be without a preceding " $ ". (This won't work in Linux, so you have to insert a " $ " before each variable in a test. Another solution is duplicate " [[ " instead of leaving " [ ] " for your test.)
Depends on the code, in my case I have found lots of variables comparing strings without being between double quotes, so we have decided to put double quotes for all variables comparing strings.
You have to change the extension of your scripts that are .ksh into .sh . Be aware that you have to modify all your cronjobs and all your scripts that are being called by other applications (like Control-M) if you still want them to function as they did.
You have to change the shebang for #!/bin/bash or #!/bin/sh or #!/usr/bin/env sh
I don't know which Linux distribution, but be aware of sh and dash for Debian/Ubuntu.
You have to change the path for commands like, I had something like this in AIX:

/usr/bin/fuser

and now it has to be changed into

/bin/fuser

Same for mailx .

We also had to look for all the executables in C and make sure it all works as they used to.
We also searched all the sqlplus calls to test all the database related things.
We had to find out a way to have which script depends on another script and which scripts are the parent scripts.
We used sh -x to execute each script to search for anything strange or any possible errors.
We made a copy of each script and then we commented each loop and test so we could execute each action of the script and see if any errors came.
tail -n +2 but without -n doesn't work in Linux.
We have found a few variables that were not declared and this was because someone forgot to change it in an old migration they did from HP-UX to AIX.
Be aware of the environment variables.
We created the whole structe in the new machine, first without caring about permissions.

I can give you more details in private.

Regards.

Chubler_XL · June 17, 2014, 5:30pm

It's been a long time since I did a big project to convert AIX scripts to RHEL. I do remember having some trouble with echo statements not expanding the backslash characters (like \n ) for example:

echo "\nStop Service now: [y]  \c"

I believe this was related to /bin/sh just being a symbolic link to bash but running in "strict compatibility" mode. However, we found that turning on shopt -s xpg_echo at the top of the script was usually enough to get the echo statements working as they used to and much less time consuming than converting everything to use printf.

Also watch out for anything that directly interacts with the /dev device files. as almost certainly they will be named differently that the AIX devices. Think here about accessing DVD/CD device or backup tape drives or changers.

Our backup scripts needed to be re-written as they used commands like sysback and tape to interact with tape libraries/changers and things were just too different under AIX, we decided rather than redevelop this it was easier to implement a commercial backup solution (I think it was CommVault), this came with an agent that managed most of the I/O and we only need to develop little stub scripts that were called to do things like stop the database or do an on-line backup to a file.

rbatte1 · June 18, 2014, 5:31am

I think I had to do a conversion of echo to print & printf

Have a good think about your DR provision too. There is no mksysb equivalent on Linux (HP has Ignite, etc.) You may have read that you just kickstart your replacement server, but you would have to get your kickstart-server recovered first (if you lose the site) and the it's only going to put you to the install state if that's how you first built it. All your patches and customisations would need to be done again, perhaps with software installs (if they write into the boot volume group such as Oracle)

It's a bigger problem to consider for Linux.

Robin

bakunin · June 18, 2014, 9:50am

Ahem, this won't work in AIX either. The only place where you can skip the "$" and have the variable expanded anyway is inside double-brackets:

var=1
(( var = var + 1 ))
print - $var

This is standard ksh and it will definitely work in Linux-ksh as well as in AIX-ksh.

The rule is: the comparison will work, regardless if the variable is quoted or not. The problem one avoids with quoting variables is this: if the variable contains whitespace, the expanded variable will lead to a syntax error once it is expanded, because the shell treats the resulting expanded value as two (or more words):

typeset var="abc"
if [ $var == abc ] ; then         # this is first expanded to
if [ abc == abc ]  ; then         # which is a legal statement

typeset var="abc def"
if [ $var == abc ] ; then                 # this is first expanded to
if [ abc def == abc ]  ; then             # which is NOT a legal statement, but
if [ "$var" == "abc" ] ; then             # this is first expanded to
if [ "abc def" == "abc" ]  ; then         # which is legal again

You might want to do that because otherwise it would be misleading, but it is not required at all. The OS only cares for the shebang and the executable flag and whcih extension the file has doesn't matter at all.

In AIX "/bin" is a link to "/usr/bin". This is one of my 7.1.3-testsystems:

# ls -l /bin
lrwxrwxrwx    1 bin      bin               8 Feb  5 09:27 /bin -> /usr/bin

According to the POSIX documents this ("-n <number>") is the modern and suggested form and should be used anyway. If at all the old syntax is only supported for backwards compatibility purposes.

I hope this helps.

bakunin

Perderabo · June 18, 2014, 10:09am

I got hit with my ksh scripts just going from RedHat 5 to RedHat 6. Yes, they changed the behavior of the builtin echo command. But they added a "-e" flag to get the old behavior. This means that a script can easily test whih echo is in operation. Just do a "echo -e" and capture the output. If you get "-e" you have the old echo. If you get no characters, you have the new echo.

If I have the old echo, I'm good. My scripts will work the way I want. But if I have the new echo, I'm in trouble. I need to arrange to add the "-e" flag to all of my echo commands. It sounds like a big deal when I explain it, but I really just add a few lines of code to each script near the top (and certainly before any echo statements)...

function myecho
{
        ''echo -e ${1:+"$@"}
        return $?
}
echo=$(echo -e)
((${#echo})) || alias echo=myecho
unset echo

Kibou · June 18, 2014, 3:30pm

bakunin:

Ahem, this won't work in AIX either. The only place where you can skip the "$" and have the variable expanded anyway is inside double-brackets:
var=1
(( var = var + 1 ))
print - $var
This is standard ksh and it will definitely work in Linux-ksh as well as in AIX-ksh.

Bakunin, it has just been tested: variables without a preceding $ work in AIX. It works with simple [ ] .

We were really surprised about this because we never saw it before. But I am talking about hundreds of scripts written like this for a bank.

At the beginning we thouhgt those scripts didn't work, but after realising that there were hundreds we started to suspect that they actually worked that way.

This is part of the intricacies of the test command, and its peculiar ways.

bakunin:

The rule is: the comparison will work, regardless if the variable is quoted or not. The problem one avoids with quoting variables is this: if the variable contains whitespace, the expanded variable will lead to a syntax error once it is expanded, because the shell treats the resulting expanded value as two (or more words):
typeset var="abc"
if [ $var == abc ] ; then         # this is first expanded to
if [ abc == abc ]  ; then         # which is a legal statement

typeset var="abc def"
if [ $var == abc ] ; then                 # this is first expanded to
if [ abc def == abc ]  ; then             # which is NOT a legal statement, but
if [ "$var" == "abc" ] ; then             # this is first expanded to
if [ "abc def" == "abc" ]  ; then         # which is legal again
You might want to do that because otherwise it would be misleading, but it is not required at all. The OS only cares for the shebang and the executable flag and whcih extension the file has doesn't matter at all.

I know that the only important thing is the shebang. Although it is only a visual matter, to me it looks like something unproperlly finished.. Anyone that sees those scripts without opening them will think that is ksh .
This is my point of view and I explain it like this: if I tell you that I have a friend and my friend's name is Pamela, what would you think? Is it a woman or a man? With the information you have, the name, you suppose that is a girl. I am sure you can "investigate" more and maybe is a boy, but someone with that name looks at first, a girl.

This case is the same, the name is information that could be used to make our life easier, to avoid the need to open the file if the name is self-explicative.

I understand your point though, but I would change the name as well.

I know it is going to be more difficult for the rest of the programs when they have to call the scripts...

The problem here is that in the scripts from AIX they used something like

/usr/bin/fuser

so I had to change it into

/bin/fuser

In the latest version of RHEL it doesn't work without the -n option. I tested it myself.

Regards.

bakunin · June 18, 2014, 3:31pm

I always preferred the equally builtin "print" over "echo". It has a lot of advantages (the "-u" switch which "echo" lacks, the "-p", ...) and no disadvantages that i know of. Original versions of the Korn shell didn't have the "echo" built in but use the external "/usr/bin/echo", which is why i started the habit of using "print" instead.

Could you please post such a script part? I administrate AIX systems for the last 25 years now (started with 3.2.3 on a RS/6000 model 32H) and i never came across such a behavior. To be honest i can't believe that.

You are right and i would change that too, no matter what. It still is not a requirement, though, and i wanted to exmphasize that difference.

bakunin

Kibou · June 19, 2014, 3:26am

The only way is testing it on an AIX machine. Do you have access yourself to an AIX machine so you can test this simple thing?

cont=1
if [ cont -eq 1 ]
then
  echo "It works"
fi

bakunin · June 19, 2014, 3:54am

I have access to the 300+ AIX systems i am in charge of, ranging from the two 5.3-systems we need to keep at that level to the latest 7.1.3 SP3 and anything in between. Today is my day off, but i will report here what i find out tomorrow.

bakunin

Kibou · June 19, 2014, 5:20am

Today is bank holiday in my country so I am not at work either...

300 AIX is a lot ^^ I just need one!

I haven't got as much experience with AIX as you have, but there's a workmate working with me that has been working with AIX for many years as well and he was shocked after testing this code.

I'll wait, no problem. Enjoy your day off!

bakunin · June 20, 2014, 4:21am

I first tried it on my NIM-Server, which happens to be also my script development system. Its version is basically 7.1 SP2. The first try is with the system shell (which is a ksh88), the second with the ksh93:

# lslpp -l bos.rte
  Fileset                      Level  State      Description         
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.rte                   7.1.2.15  COMMITTED  Base Operating System Runtime

Path: /etc/objrepos
  bos.rte                   7.1.2.15  COMMITTED  Base Operating System Runtime
# instfix -i | grep ML
    All filesets for 7.1.0.0_AIX_ML were found.
    All filesets for 7100-00_AIX_ML were found.
    All filesets for 7100-01_AIX_ML were found.
    All filesets for 7100-02_AIX_ML were found.

# Version M-11/16/88f
# cont=1
# if [ cont -eq 1 ]
>then
>  echo "It works"
>fi
It works

# ksh93
# print - ${.sh.version}
Version M 93t+ 2009-05-01
# if [ cont -eq 1 ]
>then
>  echo "It works"
>fi
It works

To be honest, i'm dumbfounded. I would have never expected that to run. Btw., this only works with integers, not with strings:

foo="abc"
if [ foo == abc ] ; then
     print - "It works"
fi

will work as expected - which is: not work at all.

I got adventurous and tried to make the datatype more "stringy" and less "integerly":

typeset -RZ3 foo=1
# typeset -RZ3 foo="01"

print - $foo
if [ foo -eq 1 ] ; then
     print - "It works"
fi

both of which works too - "foo" gets expanded.

The next thing i wondered was: is this specific to the AIX-ksh? In fact it is not. My colleague from the Linux-team tried it on a couple of our SLES11-servers and the ksh93 there worked the same. As it seems this is no specific feature of the AIX-shell but a common treat of the Korn shell.

I hope this helps.

bakunin

Kibou · June 20, 2014, 5:28am

bakunin:

I first tried it on my NIM-Server, which happens to be also my script development system. Its version is basically 7.1 SP2. The first try is with the system shell (which is a ksh88), the second with the ksh93:
# lslpp -l bos.rte
  Fileset                      Level  State      Description         
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  bos.rte                   7.1.2.15  COMMITTED  Base Operating System Runtime

Path: /etc/objrepos
  bos.rte                   7.1.2.15  COMMITTED  Base Operating System Runtime
# instfix -i | grep ML
   All filesets for 7.1.0.0_AIX_ML were found.
   All filesets for 7100-00_AIX_ML were found.
   All filesets for 7100-01_AIX_ML were found.
   All filesets for 7100-02_AIX_ML were found.

# Version M-11/16/88f
# cont=1
# if [ cont -eq 1 ]
>then
>  echo "It works"
>fi
It works

# ksh93
# print - ${.sh.version}
Version M 93t+ 2009-05-01
# if [ cont -eq 1 ]
>then
>  echo "It works"
>fi
It works
To be honest, i'm dumbfounded. I would have never expected that to run. Btw., this only works with integers, not with strings:
foo="abc"
if [ foo == abc ] ; then
   print - "It works"
fi
will work as expected - which is: not work at all.

I got adventurous and tried to make the datatype more "stringy" and less "integerly":
typeset -RZ3 foo=1
# typeset -RZ3 foo="01"

print - $foo
if [ foo -eq 1 ] ; then
   print - "It works"
fi
both of which works too - "foo" gets expanded.

The next thing i wondered was: is this specific to the AIX-ksh? In fact it is not. My colleague from the Linux-team tried it on a couple of our SLES11-servers and the ksh93 there worked the same. As it seems this is no specific feature of the AIX-shell but a common treat of the Korn shell.

I hope this helps.

bakunin

Thanks! wow so it's a korn shell issue and it behaves the same way in Linux. That's another important step..

We were completly shocked as well.

I'd like to share the story. This is what happened:

First we found out that a lot scripts were "wrong", which is very strange because is a bank..
We asked the bank, because we are outsourced, about this issue, and they said: you are right, those scripts are wrong. (lol)
Then we decided we had to fix that so those scripts work properly in RHEL now. But.. wait! if there are a lot of scripts that didn't work for years, if you now make them work.. what will it happen???
So we agreed to fix it only and just only we had the approval of all the people in charge.
The next day we got an email saying: No, don't "fix" anything at the moment and just take notes of when you found something is wrong.
As we kept going we saw more and more scripts.. hundreds.. so what's going on??
Then someone saw the light and realised that those scripts actually work. I was so shocked and puzzled I said to my manager: alright! I'm leaving! lol
So I don't know what kind of mess they have they don't even know about which script is working, etc........
The thing is that now we haven't got any more time to do those changes manually so we had to think about how to do a massive change tool to change all scripts and make them have the preceding "$" in each case, whithout messing them all up, and messing all the work we had done already. This was a problem, because it seemed kind of easy, just a matter of finding the right regular expression.. but we found out that it didn't work properly and not only worked.. but it messed up some scripts.. so we ended up using a higher level language and, instead of using regular expressions, something far less elegant but more effective: searching for the lines where there were a variable wihtout a "$", filtering it into the unique lines, and then modifiy it manually and make a file with the origin and the target, separated with "@@@" (as an example). We used a higher level language to make the substitution in all scripts and it worked perfectly except for 200 scripts, with just scripts that didn't work.
We have been working monday to sunday lately..

So now I believe in God again lol!

Chubler_XL · June 22, 2014, 4:50pm

bakunin:

I got adventurous and tried to make the datatype more "stringy" and less "integerly":
typeset -RZ3 foo=1
# typeset -RZ3 foo="01"

print - $foo
if [ foo -eq 1 ] ; then
   print - "It works"
fi
both of which works too - "foo" gets expanded.

Yes, even these seem to work:

foo=8#14
[ foo -eq 12 ] && print - "It works"

foo=21+3-0x0A
[ foo -eq 14 ] && print - "It works"

bakunin · June 23, 2014, 3:00am

Addendum: i got me a Ubuntu system and the code works there too. This seems to be a ksh issue rather than an AIX issue:

# echo ${.sh.version}
Version JM 93t+ 2010-03-05
# foo=1
# if [ foo -eq 1 ] ; then echo "works" ; fi
works

Systems tested to behave in the described manner so far:

AIX
SLES 11
Ubuntu 10.04 LTS

bakunin

CarloM · June 23, 2014, 8:05am

Also Solaris 10:

$ uname -rs
SunOS 5.10
$ foo=1
$ if [ foo -eq 1 ] ; then echo "works" ; fi
works

Don_Cragun · June 24, 2014, 1:57am

I was asked to take a look at this and comment on what the standards say about this ksh behavior. The standards leave unspecified what happens if both operands of the test -eq , -ne , -lt , -le , -gt , and -ge binary primaries are not integer operands.

The standards allow, but do not require the test expression and [ expression ] utilities to be implemented as shell built-ins (as they are in bash and ksh )). If test and [ ... ] aren't implemented as shell built-ins, there is no way that the behavior shown here would be possible. In ksh (at least ksh on AIX and on OS X), if a variable name is given where a number is needed in these cases, the value of the variable is substituted into the expression even when the $ is not present.

The standards specify this behavior for arithmetic expansions. That is, the sequences:

n=0
while [ $n -lt 5 ]
do      printf "%d\n" $(($n++))
done

and:

n=0
while [ $n -lt 5 ]
do      printf "%d\n" $((n++))
done

are required to do the same thing. So, it seems like a logical extension to me to not require the $ in the expression in these test exp ( [ exp ] ) expressions either.

Obviously, this need not be portable to other shells and might not even be portable to other versions of ksh .