Problem evaluating condition

First, given this bit of code (line numbers included for reference:

 59 get_all_db () {
 60 #echo getting all db
 61 dblist=`egrep -i "product/12" /etc/oratab |grep -v "listener"|\
 62       awk -F\: '{print $1}'|sort`
 63 echo list is $dblist
 64 echo
 65 echo
 66 echo "INSTANCE_NAME    HTTPS Port  HTTP Port"
 67 echo "---------------- ---------- ----------"
 68 for ORACLE_SID in $dblist
 69 do
 70 echo checking for pmon_$ORACLE_SID
 71 ps -ef|grep pmon_$ORACLE_SID|grep -v grep
 72 echo ps command complete
 73   if ps -ef|grep pmon_$ORACLE_SID|grep -v grep
 74   then
 75     echo calling sqlplus
 76     . oraenv
 77     sqlplus -s /nolog <<EOF
 78     set echo off feedback off verify off head off
 79     conn / as sysdba
 80     select instance_name,
 81            DBMS_XDB_CONFIG.gethttpsport "HTTPS Port",
 82            DBMS_XDB_CONFIG.gethttpport "HTTP Port"
 83     FROM v\$instance;
 84 EOF
 85   fi
 86 done
 87 }

Producing this output:

  1 list is db12 fs91upg fs92dev fs92upg hr92dev hr92tst hr92upg
  2
  3
  4 INSTANCE_NAME    HTTPS Port  HTTP Port
  5 ---------------- ---------- ----------
  6 checking for pmon_db12
  7 ps command complete
  8 checking for pmon_fs91upg
  9 ps command complete
 10 checking for pmon_fs92dev
 11 ps command complete
 12 checking for pmon_fs92upg
 13 ps command complete
 14 checking for pmon_hr92dev
 15 ps command complete
 16 checking for pmon_hr92tst
 17 ps command complete
 18 checking for pmon_hr92upg
 19 ps command complete

What I don't understand is why the IF continues to evaluate FALSE. The first iteration of the DO loop, where the value of $ORACLE_SID is 'db12' should evaluate false, but the rest are true. To help debug I added lines 70 and 71 to show the current value of ORACLE_SID and the result of the same command that produces the result to be evaluated by the IF. You can see from the output that it appears to always return null (false) but if I execute the very same command manually, it returns the expected result:

  1 oracle:fs91upg$ echo $ORACLE_SID
  2 fs91upg
  3
  4 oracle:fs91upg$ ps -ef|grep pmon_$ORACLE_SID|grep -v grep
  5 oracle    2842     1  0 Jan22 ?        00:00:34 ora_pmon_fs91upg
  6
  7 2018-01-26 08:59:44
  8 oracle:fs91upg$

I may say load of nonsense as just got home not too well and very tired so forgive me...
What I see is what you say give what you expected for an output is a command line:

 ps -ef|grep pmon_$ORACLE_SID|grep -v grep 

What is it made of? I see 3 processes, and yes nothing wrong there so you get what is expected but
looking at your if, what do you think is the condition?

So what about replacing your if by :

if  [ -n  "$(ps -ef|grep pmon_$ORACLE_SID|grep -v grep)" ]  

or the correct variant ( sorry can't test, and in bed now...)

Addendum:
That is what I would do in ksh...

First, let me point out that I've edited my opening post, to add reference line numbers to all 'code' blocks. Herein I use the term 'code block' to refer to any section of my posts that are set off as 'code', regardless of whether it is actual script code, or other type of output taken from the shell session.

In the posted script, line 71 (the 'ps' command) should echo to stdout exactly what is being tested with the IF at line 73. The expected output for everything except the first iteration is shown in the last 'code' block in my opening post. For example, on the second iteration of the DO loop, the value of ORACLE_SID is 'fs91upg'. And we see this in the second code block, at line 8, as echoed from line 70 of the script. But we do not see any output from script line 71. That same command - line 71 of the script - produces the output shown at line 5 of the third 'code' block.

Of course, the fact that script line 71 never produces any output explains why the IF on line 73 is evaluating FALSE. So the question is not really about the IF, per se, but why that 'ps' command (both at script lines 71 and 73 is returning nulls when the very same command, with the very same values, returns expected data when issued from the command prompt.

I'm afraid that using the result of a "command substitution" for an if condition is not too good an idea, as the shell it will try to run the result line as a command which will most probably fail.
What makes me somewhat hesitant is that

ps -ef|grep pmon_$ORACLE_SID|grep -v grep 

doesn't seem to produce a result as can be seen in the output in lines 8 and 9. grep not producing an output line - and be it through the -v option - will yield the exit code 1 which is interpreted as FALSE.
Please show the result of the command

ps -ef|grep pmon_$ORACLE_SID

in your function. Is it possible the $ORACLE_SID has non-printing control chars in it?

I woke up and corrected my if the way I would do that with ksh...

I agree with RudiC that there doesn't appear to be anything obvious. Please add the following line of code between lines 86 and 87 in your get_all_db() function:

set +xv

then add the line:

set -xv

between lines 59 and 60 in that same function, then run your script again, and show us the output produced.

You could also replace:

 ps -ef|grep pmon_$ORACLE_SID|grep -v grep

in both places with:

 ps -ef|grep '[p]'mon_$ORACLE_SID

but this should only make the code run faster; not affect the output produced.

Rudic -
To show the "raw" value of $ORACLE_SID at a command line:

oracle:hr92upg$ echo $ORACLE_SID
hr92upg

And to show no hidden characters, I'll append an 'x' on each end:

oracle:hr92upg$ echo x${ORACLE_SID}x
xhr92upgx

And, as requested, the ps and grep, without the -v to filter out the grep itself

oracle:hr92upg$ ps -ef|grep pmon_$ORACLE_SID
oracle    1237     1  0 10:37 ?        00:00:01 ora_pmon_hr92upg
oracle   31557 28288  0 12:59 pts/0    00:00:00 grep pmon_hr92upg

And in the interest of full disclosure, all possible 'pmon' processes:

oracle:hr92upg$ ps -ef|grep pmon
oracle    1237     1  0 10:37 ?        00:00:01 ora_pmon_hr92upg
oracle    1631     1  0 Jan22 ?        00:00:46 ora_pmon_fs91dmo
oracle    2842     1  0 Jan22 ?        00:00:35 ora_pmon_fs91upg
oracle    4317     1  0 Jan22 ?        00:00:49 ora_pmon_fs91dev
oracle    9786     1  0 Jan04 ?        00:03:03 ora_pmon_fs92upg
oracle   10156     1  0 Jan04 ?        00:03:02 ora_pmon_hr92dev
oracle   21794     1  0 Jan05 ?        00:02:58 ora_pmon_fs92dev
oracle   23261     1  0 Jan25 ?        00:00:11 ora_pmon_hr92tst
oracle   31578 28288  0 12:59 pts/0    00:00:00 grep pmon

Compare the above with the list of possible values for ORACLE_SID generated at line 61 of my script, and displayed at line 63

Strange. Did you run

ps -ef|grep pmon_$ORACLE_SID

within your function? To show e.g. control chars,

echo $ORACLE_SID | od -tx1c
0000000  68  72  39  32  75  70  67  0a
          h   r   9   2   u   p   g  \n

Do this within your function!
And, as Don Cragun proposed, run parts of your function with the -vx set.

Don Cragun -
Adding the 'set' commands as requested, here is the output (line numbers added for reference).

 1 egrep -i "product/12" /etc/oratab |grep -v "listener"|      awk -F\: '{print     $1}'|sort
  2 ++ egrep -i product/12 /etc/oratab
  3 ++ grep -v listener
  4 ++ awk -F: '{print $1}'
  5 ++ sort
  6 + dblist='db12
  7 fs91upg
  8 fs92dev
  9 fs92upg
 10 hr92dev
 11 hr92tst'
 12 + echo list is db12 fs91upg fs92dev fs92upg hr92dev hr92tst
 13 list is db12 fs91upg fs92dev fs92upg hr92dev hr92tst
 14 + echo
 15
 16 + echo
 17
 18 + echo 'INSTANCE_NAME    HTTPS Port  HTTP Port'
 19 INSTANCE_NAME    HTTPS Port  HTTP Port
 20 + echo '---------------- ---------- ----------'
 21 ---------------- ---------- ----------
 22 + for ORACLE_SID in '$dblist'
 23 + echo checking for pmon_db12
 24 checking for pmon_db12
 25 + ps -ef
 26 + grep pmon_db12
 27 + grep -v grep
 28 + echo ps command complete
 29 ps command complete
 30 + ps -ef
 31 + grep pmon_db12
 32 + grep -v grep
 33 + for ORACLE_SID in '$dblist'
 34 + echo checking for pmon_fs91upg
 35 checking for pmon_fs91upg
 36 + ps -ef
 37 + grep pmon_fs91upg
 38 + grep -v grep
 39 + echo ps command complete
 40 ps command complete
 41 + ps -ef
 42 + grep pmon_fs91upg
 43 + grep -v grep
 44 + for ORACLE_SID in '$dblist'
 45 + echo checking for pmon_fs92dev
 46 checking for pmon_fs92dev
 47 + ps -ef
 48 + grep pmon_fs92dev
 49 + grep -v grep
 50 + echo ps command complete
 51 ps command complete
 52 + ps -ef
 53 + grep pmon_fs92dev
 54 + grep -v grep
 55 + for ORACLE_SID in '$dblist'
 56 + echo checking for pmon_fs92upg
 57 checking for pmon_fs92upg
 58 + ps -ef
 59 + grep pmon_fs92upg
 60 + grep -v grep
 61 + echo ps command complete
 62 ps command complete
 63 + ps -ef
 64 + grep -v grep
 65 + grep pmon_fs92upg
 66 + for ORACLE_SID in '$dblist'
 67 + echo checking for pmon_hr92dev
 68 checking for pmon_hr92dev
 69 + ps -ef
 70 + grep pmon_hr92dev
 71 + grep -v grep
 72 + echo ps command complete
 73 ps command complete
 74 + ps -ef
 75 + grep pmon_hr92dev
 76 + grep -v grep
 77 + for ORACLE_SID in '$dblist'
 78 + echo checking for pmon_hr92tst
 79 checking for pmon_hr92tst
 80 + ps -ef
 81 + grep pmon_hr92tst
 82 + grep -v grep
 83 + echo ps command complete
 84 ps command complete
 85 + grep pmon_hr92tst
 86 + ps -ef
 87 + grep -v grep
 88 + set +xv

As I mentioned in an earlier message, at this point I'm not so much concerned about the failure of the IF condition. What's more puzzling to me at this point is the failure of lines 70-71 of the function. The output of line 70 is clearly looping through the possible values of ORACLE_SID, as driven by $dblist, but the ps -ef|grep pmon_$ORACLE_SID is not getting a hit, even though the processes are there (see my response to Rudi)

---------- Post updated at 02:08 PM ---------- Previous update was at 01:21 PM ----------

---------- Post updated at 02:10 PM ---------- Previous update was at 02:08 PM ----------

RudiC - I think you nailed it.

Modified the DO loop as follows. Note lines 71-74

 69 for ORACLE_SID in $dblist
 70 do
 71   echo checking for pmon_$ORACLE_SID
 72   echo $ORACLE_SID | od -tx1c
 73   ps -ef|grep pmon_$ORACLE_SID|grep -v grep
 74   echo ps command complete
 75   if ps -ef|grep pmon_$ORACLE_SID|grep -v grep
 76   then
 77     echo calling sqlplus
 78     . oraenv
 79     sqlplus -s /nolog <<EOF
 80     set echo off feedback off verify off head off
 81     conn / as sysdba
 82     select instance_name,
 83            DBMS_XDB_CONFIG.gethttpsport "HTTPS Port",
 84            DBMS_XDB_CONFIG.gethttpport "HTTP Port"
 85     FROM v\$instance;
 86 EOF
 87   fi
 88 done
 89 #set +xv

And the result:

list is db12 fs91upg fs92dev fs92upg hr92dev hr92tst


INSTANCE_NAME    HTTPS Port  HTTP Port
---------------- ---------- ----------
checking for pmon_db12
0000000  64  62  31  32  0a
          d   b   1   2  \n
0000005
ps command complete
checking for pmon_fs91upg
0000000  66  73  39  31  75  70  67  0a
          f   s   9   1   u   p   g  \n
0000010
ps command complete
checking for pmon_fs92dev
0000000  66  73  39  32  64  65  76  0a
          f   s   9   2   d   e   v  \n
0000010
ps command complete
checking for pmon_fs92upg
0000000  66  73  39  32  75  70  67  0a
          f   s   9   2   u   p   g  \n
0000010
ps command complete
checking for pmon_hr92dev
0000000  68  72  39  32  64  65  76  0a
          h   r   9   2   d   e   v  \n
0000010
ps command complete
checking for pmon_hr92tst
0000000  68  72  39  32  74  73  74  0a
          h   r   9   2   t   s   t  \n
0000010
ps command complete
 

So, as the list of ORACLE_SID values is generated in the script, they are getting a x'0D'. So how do I strip that out?

The 0x0D (= <CR> , \r , ^M ) would perfectly back my theory, but where do you see it? I can't, in your post.

Howsoever, as you extract the ORACLE_SIDs from /etc/oratab that might be the place to start from. How was it created? With some MS editor? That would explain the 0x0D as a line terminator. Is it possible to edit the file with a native *nix editor, eliminating the <CR> ?
Or, your dblist definition might be rewritten like (untested)

dblist=$(awk -F\: '
toupper($0) ~ /PRODUCT\/12/ &&
! /listener/    {gsub ("\r", ""); print $1}
' /etc/oratab | sort)

Having carriage return characters in the ORACLE_SID values would explain the results you're getting. If you don't find the carriage return characters in the ORACLE_SID values, the next step would be to look for aliases for the utilities you're using in your function.

Outside of your script, type the following commands into your shell:

type echo grep ps
uname -n
ORACLE_SID=fs
ps -ef|grep '[p]'mon_$ORACLE_SID

Then add those same commands into your function just after the set -xv line and show us the output from both of them.

RudiC -
I'm sorry, not x0D but x 0A. It's in the output I posted.

As for the format of /etc/oratab, no it is not created with Windows editor. This is pure linux, and /etc/oratab is a simple text file. Sometimes modified by Oracle utilites, sometimes manually using vi. It looks like this:

oracle:listener$ cat /etc/oratab
#
# This file is used by ORACLE utilities.  It is created by root.sh
# and updated by the Database Configuration Assistant when creating
# a database.
#
# A colon, ':', is used as the field terminator.  A new line terminates
# the entry.  Lines beginning with a pound sign, '#', are comments.
#
# Entries are of the form:
#   $ORACLE_SID:$ORACLE_HOME:<N|Y>:
#
# The first and second fields are the system identifier and home
# directory of the database respectively.  The third filed indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#============================================
db11:/u01/app/oracle/product/11.2.0.4/dbhome_1:N
db12:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
listener:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
fs91dmo:/u01/app/oracle/product/11.2.0.4/dbhome_1:N
fs91dev:/u01/app/oracle/product/11.2.0.4/dbhome_1:N
hr92dev:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
fs92upg:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
fs92dev:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
hr92tst:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
fs91upg:/u01/app/oracle/product/12.1.0.2/dbhome_1:N
hr92upg:/u01/app/oracle/product/11.2.0.4/dbhome_1:N

---------- Post updated at 07:35 AM ---------- Previous update was at 07:25 AM ----------

Don -

Requested output follows. Output of uname obfuscated for security.

oracle:hr92dev$ type echo grep ps
echo is a shell builtin
grep is /bin/grep
ps is /bin/ps

oracle:hr92dev$ uname -n
myserver.myorg.org

oracle:hr92dev$ ORACLE_SID=fs

oracle:fs$ ps -ef|grep '[p]'mon_$ORACLE_SID
oracle    1631     1  0 Jan22 ?        00:00:54 ora_pmon_fs91dmo
oracle    2842     1  0 Jan22 ?        00:00:41 ora_pmon_fs91upg
oracle    4317     1  0 Jan22 ?        00:00:57 ora_pmon_fs91dev
oracle    9786     1  0 Jan04 ?        00:03:09 ora_pmon_fs92upg
oracle   21794     1  0 Jan05 ?        00:03:05 ora_pmon_fs92dev


+ type echo grep ps
echo is a shell builtin
grep is /bin/grep
ps is /bin/ps
+ uname -n
myserver.myorg.org
+ ORACLE_SID=fs
+ ps -ef
+ grep '[p]mon_fs'
oracle    1631     1  0 Jan22 ?        00:00:54 ora_pmon_fs9
oracle    2842     1  0 Jan22 ?        00:00:42 ora_pmon_fs9
oracle    4317     1  0 Jan22 ?        00:00:57 ora_pmon_fs9
oracle    9786     1  0 Jan04 ?        00:03:09 ora_pmon_fs9
oracle   21794     1  0 Jan05 ?        00:03:05 ora_pmon_fs9
egrep -i "product/12" /etc/oratab |grep -v "listener"|      awk -F\: '{print $1}'|sort
++ egrep -i product/12 /etc/oratab
++ grep -v listener
++ awk -F: '{print $1}'
++ sort
+ dblist='db12
fs91upg
fs92dev
fs92upg
hr92dev
hr92tst'
+ echo list is db12 fs91upg fs92dev fs92upg hr92dev hr92tst
list is db12 fs91upg fs92dev fs92upg hr92dev hr92tst
+ echo

+ echo

+ echo 'INSTANCE_NAME    HTTPS Port  HTTP Port'
INSTANCE_NAME    HTTPS Port  HTTP Port
+ echo '---------------- ---------- ----------'
---------------- ---------- ----------
+ for ORACLE_SID in '$dblist'
+ echo checking for pmon_db12
checking for pmon_db12
+ echo db12
+ od -tx1c
0000000  64  62  31  32  0a
          d   b   1   2  \n
0000005
+ ps -ef
+ grep pmon_db12
+ grep -v grep
+ echo ps command complete
ps command complete
+ ps -ef
+ grep pmon_db12
+ grep -v grep
+ for ORACLE_SID in '$dblist'
+ echo checking for pmon_fs91upg
checking for pmon_fs91upg
+ echo fs91upg
+ od -tx1c
0000000  66  73  39  31  75  70  67  0a
          f   s   9   1   u   p   g  \n
0000010
+ ps -ef
+ grep pmon_fs91upg
+ grep -v grep
+ echo ps command complete
ps command complete
+ ps -ef
+ grep pmon_fs91upg
+ grep -v grep
+ for ORACLE_SID in '$dblist'
+ echo checking for pmon_fs92dev
checking for pmon_fs92dev
+ echo fs92dev
+ od -tx1c
0000000  66  73  39  32  64  65  76  0a
          f   s   9   2   d   e   v  \n
0000010
+ ps -ef
+ grep pmon_fs92dev
+ grep -v grep
+ echo ps command complete
ps command complete
+ ps -ef
+ grep pmon_fs92dev
+ grep -v grep
+ for ORACLE_SID in '$dblist'
+ echo checking for pmon_fs92upg
checking for pmon_fs92upg
+ echo fs92upg
+ od -tx1c
0000000  66  73  39  32  75  70  67  0a
          f   s   9   2   u   p   g  \n
0000010
+ ps -ef
+ grep pmon_fs92upg
+ grep -v grep
+ echo ps command complete
ps command complete
+ ps -ef
+ grep pmon_fs92upg
+ grep -v grep
+ for ORACLE_SID in '$dblist'
+ echo checking for pmon_hr92dev
checking for pmon_hr92dev
+ echo hr92dev
+ od -tx1c
0000000  68  72  39  32  64  65  76  0a
          h   r   9   2   d   e   v  \n
0000010
+ ps -ef
+ grep -v grep
+ grep pmon_hr92dev
+ echo ps command complete
ps command complete
+ ps -ef
+ grep pmon_hr92dev
+ grep -v grep
+ for ORACLE_SID in '$dblist'
+ echo checking for pmon_hr92tst
checking for pmon_hr92tst
+ echo hr92tst
+ od -tx1c
0000000  68  72  39  32  74  73  74  0a
          h   r   9   2   t   s   t  \n
0000010
+ ps -ef
+ grep pmon_hr92tst
+ grep -v grep
+ echo ps command complete
ps command complete
+ ps -ef
+ grep pmon_hr92tst
+ grep -v grep
+ set +xv

I'm afraid I'm out of ideas. Do you have the shell variable GREP_OPTIONS set? You may want to run an unmodified, pure ps -ef in your function just to see what it prints.

Hold on, hold on! How do you explain the difference in above outputs:

oracle:fs$ ps -ef|grep '[p]'mon_$ORACLE_SID
oracle    1631     1  0 Jan22 ?        00:00:54 ora_pmon_fs91dmo
oracle    2842     1  0 Jan22 ?        00:00:41 ora_pmon_fs91upg
oracle    4317     1  0 Jan22 ?        00:00:57 ora_pmon_fs91dev
oracle    9786     1  0 Jan04 ?        00:03:09 ora_pmon_fs92upg
oracle   21794     1  0 Jan05 ?        00:03:05 ora_pmon_fs92dev


+ type echo grep ps
echo is a shell builtin
grep is /bin/grep
ps is /bin/ps
+ uname -n
myserver.myorg.org
+ ORACLE_SID=fs
+ ps -ef
+ grep '[p]mon_fs'
oracle    1631     1  0 Jan22 ?        00:00:54 ora_pmon_fs9
oracle    2842     1  0 Jan22 ?        00:00:42 ora_pmon_fs9
oracle    4317     1  0 Jan22 ?        00:00:57 ora_pmon_fs9
oracle    9786     1  0 Jan04 ?        00:03:09 ora_pmon_fs9
oracle   21794     1  0 Jan05 ?        00:03:05 ora_pmon_fs9

If the second one is from within the function, you'll never find a match with your ORACLE_SID. Was the COLUMNS shell variable modified? The second output is clipped at 60 chars... strange

It looks like RudiC has identified the source of your problem.

I'm glad that my idea of comparing internal and external invocations of ps showed us that something in your script is affecting the output from ps .

So, try running the command:

grep COLUMNS scriptname

where scriptname is a pathname to your shell script. And, add:

echo COLUMNS is $COLUMNS

to the list of things to run inside your function and outside your script to verify that we have identified the problem.

When you obfuscated the output of the uname -n commands, you may have hidden what I was looking for. I don't care about the name of your system, but I do want you to verify that the output from both invocations of uname produced exactly the same output. (We need to be sure that there isn't an ssh or something in your script that is causing some of the code to be executed on a different server!)

Bingo! The code that called the function looked like this:

ORACLE_SID=''
PS3='Select  database: '
savcol=$COLUMNS
export COLUMNS=20
#
while [[ $ORACLE_SID = "" ]]; do
  select ORACLE_SID in `egrep -i "product/12" /etc/oratab |grep -v "listener"|\
      awk -F\: '{print $1}'|sort` "All of the above" "None of the above" ; do
    if [[ $ORACLE_SID = "" ]]; then
         echo
         echo "Please enter a valid number.  Retry.";
         echo
    elif [[ $ORACLE_SID = "None of the above" ]]; then
         exit ;
    elif [[ $ORACLE_SID = "All of the above" ]]; then
         get_all_db;
    else {
          get_one_db;
         }
    fi
    break
    done
done
#
unset PS3
export COLUMNS=$savcol

It was based on code from a previous script, where the IF in the loop was just setting values to be acted on after exiting the loop. In this script I decided to create functions to be executed directly from the IF statement, and in that forgot about the need to reset the COLUMNS. I added that statement into the IF construct, just before calling the functions:

    elif [[ $ORACLE_SID = "All of the above" ]]; then
         export COLUMNS=$savcol;
         get_all_db;
    else {
          export export COLUMNS=$savcol;
          get_one_db;
         }
    fi

and all is working correctly.

Thanks for all the assistance, and patience.