ksh behavior change on RHEL5

I recently patched a RHEL5 host from 5.2 to 5.5. There are some scripts that simply do a ps and grep to check for itself before proceeding. I personally don't like the way the scripts are written and would have done it differently. Before I can proceed with patching the production servers, I need to determine why the behaviour changed with these scripts. I've provided a sample script that reproduces the issue and output from both the patched system and a none patched system. Running the script manually everything works as expected, but these run through cron and give different output.

A small oddity I just noticed too, when type 'cat /etc/redhat' and hitting tab on the patched system you can see below that it put a '\' before the '-', but on the unpatched system it does not.

Crontab:

* * * * * /app2/stage/aim/aim_ap_etl/test.ksh > /app2/stage/aim/aim_ap_etl/test.out 2>&1

Script:

#!/bin/ksh

echo $0
SCRIPT=${0##*/}
echo $SCRIPT
echo "IFS=xx${IFS}xx"
echo "PS=$(which ps)"
echo "GREP=$(which grep)"
echo "PGREP=$(which pgrep)"
echo "WC=$(which wc)"
echo "Running processes: "
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT
/bin/ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT | /usr/bin/wc -l
/usr/bin/pgrep $SCRIPT | /usr/bin/wc -l
echo "-----------"

proc1=`ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT`
echo "proc1=$proc1"

proc2=$(ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT)
echo "proc2=$proc2"

proc3=$(/usr/bin/pgrep $SCRIPT | /usr/bin/wc -l)
echo "proc3=$proc3"

echo "Exiting!"
exit

Non-patched

> uname -a
Linux xxx 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64

> cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.2 (Tikanga)

> echo ${.sh.version}
Version M 1993-12-28 r

Manual output with ksh:

> ./test.ksh
./test.ksh
test.ksh
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes: 
infa      2365 24775  0 12:28 pts/2    00:00:00 /bin/ksh ./test.ksh
1
1
1
-----------
proc1=1
proc2=1
proc3=1
Exiting!

Manual output with bash:

> ./test.bash
./test.bash
test.bash
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes: 
infa      2914 24775  0 12:30 pts/2    00:00:00 /bin/bash ./test.bash
1
1
1
-----------
proc1=2
proc2=2
proc3=2
Exiting!

Crontab output with ksh:

> cat test.out
/app2/stage/aim/aim_ap_etl/test.ksh
test.ksh
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes: 
infa      3823  3820  0 12:36 ?        00:00:00 /bin/sh -c /app2/stage/aim/aim_ap_etl/test.ksh > /app2/stage/aim/aim_ap_etl/test.out 2>&1
infa      3824  3823  0 12:36 ?        00:00:00 /bin/ksh /app2/stage/aim/aim_ap_etl/test.ksh
2
2
1
-----------
proc1=2
proc2=2
proc3=1
Exiting!

Patched

> uname -a
Linux xxx 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

> cat /etc/redhat\-release                                                                               
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

> echo ${.sh.version}
Version AJM 93t+ 2010-02-02

Manual output with ksh:

> ./test.ksh
./test.ksh
test.ksh
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes: 
infa      3288 11352  0 12:27 pts/3    00:00:00 /bin/ksh ./test.ksh
1
1
1
-----------
proc1=2
proc2=2
proc3=2
Exiting!

Manual output with bash:

> ./test.bash
./test.bash
test.bash
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes: 
infa      3702 11352  0 12:28 pts/3    00:00:00 /bin/bash ./test.bash
1
1
1
-----------
proc1=2
proc2=2
proc3=2
Exiting!

Crontab output with ksh:

> cat test.out
/app2/stage/aim/aim_ap_etl/test.ksh
test.ksh
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes: 
infa      3962  3961  0 12:32 ?        00:00:00 /bin/sh -c /app2/stage/aim/aim_ap_etl/test.ksh > /app2/stage/aim/aim_ap_etl/test.out 2>&1
infa      3963  3962  0 12:32 ?        00:00:00 /bin/ksh /app2/stage/aim/aim_ap_etl/test.ksh
2
2
1
-----------
proc1=3
proc2=3
proc3=2
Exiting!

Can you log and report to us what value ksh thinks the SCRIPT variable has when you run the cron job?

It's the second value echo'd in the script.

echo $SCRIPT

test.ksh

---------- Post updated at 13:24 ---------- Previous update was at 13:21 ----------

Also, any idea why bash outputs a different value than ksh? Most of the values are 2 instead of 1.

I run script consecutively for testing..

[root@rhnserver ~]# ./script
./script
script
IFS=xx
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes:
root     32647 30389  0 05:03 pts/0    00:00:00 /bin/ksh ./script
1
1
1
-----------
proc1=1
proc2=1
proc3=1
Exiting!
[root@rhnserver ~]# ./script
./script
script
IFS=xx
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
Running processes:
root     32672 30389  0 05:03 pts/0    00:00:00 /bin/ksh ./script
1
1
1
-----------
proc1=1
proc2=2
proc3=1
Exiting!

The result is different,because it works same time cron
(let think the cron every minute works )
And cron is seeing both your script both bash and ksh

Try diffrent process for example
"script=sshd"
Must be no different the output

The cron was not running at the same time. I was done afterwards. The value expected from cron is 2 because cron spawns the script using Bourne shell (sh) as seen from the output.

infa 3962 3961 0 12:32 ? 00:00:00 /bin/sh -c /app2/stage/aim/aim_ap_etl/test.ksh > /app2/stage/aim/aim_ap_etl/test.out 2>&1
infa 3963 3962 0 12:32 ? 00:00:00 /bin/ksh /app2/stage/aim/aim_ap_etl/test.ksh

The primary issue that I'm dealing with is that the developers who created a script that runs on these hosts are producing different output since the patching. As you can see, the patched system is assigning the value of 3 when it should be 2 to the proc variables when run via cron.

I dont think a cron related problem..Also manuel outputs is giving same result that add one more to variables on patched systems..
Maybe when your ps value send to variable kernel show another process that named called script..

Maybe

root     14760 14747  0 06:43 pts/1    00:00:00 [script]

And you try probably `ps aux` for clear result..

Or you must put same lines after variables again in your code
Again lets see output

 
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT | /usr/bin/wc -l

echo "-----------"
 
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT
proc1=`ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT`
echo "proc1=$proc1"
 
/bin/ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT
proc2=$( (ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT) )
echo "proc2=$proc2"
 
/usr/bin/pgrep $SCRIPT | /usr/bin/wc -l
proc3=$(/usr/bin/pgrep $SCRIPT | /usr/bin/wc -l)
echo "proc3=$proc3"

/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT | /usr/bin/wc -l

You're correct, cron is not the issue specifically. It's the fact that the assigned value is one more than it should be.

#!/bin/ksh

echo $0
SCRIPT=${0##*/}
echo $SCRIPT
echo "IFS=xx${IFS}xx"
echo "PS=$(which ps)"
echo "GREP=$(which grep)"
echo "PGREP=$(which pgrep)"
echo "WC=$(which wc)"

/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT | /usr/bin/wc -l

echo "-----------"
 
/bin/ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT
proc1=`/bin/ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT`
echo "proc1=$proc1"
 
echo "-----------"
/bin/ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT
proc2=$( (ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT) )
echo "proc2=$proc2"
 
echo "-----------"
/usr/bin/pgrep $SCRIPT | /usr/bin/wc -l
proc3=$(/usr/bin/pgrep $SCRIPT | /usr/bin/wc -l)
echo "proc3=$proc3"

echo "-----------"
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT | /usr/bin/wc -l

output from patched

> ./test2.ksh
./test2.ksh
test2.ksh
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
brian     3728   553  0 15:48 pts/8    00:00:00 /bin/ksh ./test2.ksh
1
-----------
1
proc1=2
-----------
1
proc2=2
-----------
1
proc3=2
-----------
brian     3728   553  0 15:48 pts/8    00:00:00 /bin/ksh ./test2.ksh
1

output from non-patched

> ./test2.ksh
./test2.ksh
test2.ksh
IFS=xx 
xx
PS=/bin/ps
GREP=/bin/grep
PGREP=/usr/bin/pgrep
WC=/usr/bin/wc
brian    11485 10981  0 15:50 pts/1    00:00:00 /bin/ksh ./test2.ksh
1
-----------
1
proc1=1
-----------
1
proc2=1
-----------
1
proc3=1
-----------
brian    11485 10981  0 15:50 pts/1    00:00:00 /bin/ksh ./test2.ksh
0

The last number of the non-patched system is incorrect here. "0" should be "1"

I believe the discrepancy is due to the inner subshells, i.e. $( ... ) appearing with the same ps output command line as their parent script with later ksh93. One way to filter them out would be to use their parent pid.

Everything you are seeing is what happens when shell scripts depend on shell implementation details which are not prescribed by any standard.

I don't use RHEL and I only have one version of ksh with which to test, but I'm almost certain that the differences that you see between the old and new system is due to a ksh version update and a change in the way those versions execute pipelines/subshells.

If you take a look at the posix standard, it specifies the details of the execution environment, but it says nothing regarding implementation. The reason the command substitutions often yield a number 1 greater than the value from the same command that is not part of a substitution is because the shell has forked another copy of itself to execute the pipeline, and that new copy has the same $0.

The reason you see that 0 in your most recent post is because the shell has most likely exec'd into one of the utilities in its final pipeline before the ps utility fetches the process data. If you add another command after that pipeline, you'll probably see the count remain at 1.

In any case, these scripts make unportable assumptions. Not only are they not portable between different shells, they are not even portable between different versions of the same shell (as you have no doubt noticed ;)).

The best course of action (unless there is a massively immovable obstacle in the way) would be to rewrite these to not depend on the low level implementation of the shell.

Regards,
Alister

---------- Post updated at 07:15 PM ---------- Previous update was at 06:19 PM ----------

I can reproduce that behavior with the following script (pswc.sh):

#!/bin/ksh

ktrace -tc -ip $$
echo $0
SCRIPT=${0##*/}
echo $SCRIPT
/bin/ps | grep -v grep | grep $SCRIPT
/bin/ps | grep -v grep | grep $SCRIPT | /usr/bin/wc -l

KSH Version:

$ ksh --version
  version         sh (AT&T Labs Research) 1993-12-28 p

Output:

# ./pswc.sh 
./pswc.sh
pswc.sh
 1388  p1  S+     0:00.01 /bin/ksh ./pswc.sh
       0

Also, the trace output confirms that by the time ps starts to read the process list data, all of the ksh instances are gone (exec'd into greps, ps, and wc).

There is nothing "wrong" here, except the expectations of the script.

Hope that helps,
Alister

P.S. I can attach the trace output, but I doubt it's of much interest. It's not a RHEL system (it's not even linux), and you can probably reproduce it yourself with strace (or whatever is the name of the linux equivalent of ktrace).

I agree. I do not like the way that the "developers" write their shell scripts. I'm going to recommend that they change the way they are doing it and move on. I think they were simply counting on an old bug in the previous version.

Thanks for all the feedback.

Not a bug but undefined behavior. The bug was in the script, not in the ksh implementation.

Reason is seem like pathces..Allright..let try as last

echo $0
SCRIPT=${0##*/}
echo $SCRIPT
echo "IFS=xx${IFS}xx"
echo "PS=$(which ps)"
echo "GREP=$(which grep)"
echo "PGREP=$(which pgrep)"
echo "WC=$(which wc)"
 
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT
/bin/ps -ef | /bin/grep -v grep | /bin/grep $SCRIPT | /usr/bin/wc -l
echo "-----------"
/bin/ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT
/bin/ps -ef | /bin/grep -v grep | /bin/grep -c $SCRIPT > procx
echo "proc1=$(cat procx)"
echo "-----------"

or the simpler:

pgrep $SCRIPT | wc -l > procx
echo "proc1=$(<procx)"