Hello Guys,
I was trying to upgrade the PowerHA from 6.1.0.9 to 7.1.3 in prod cluster. Have snapshot migration and run climgcheck.
- Run climgcheck on node1(kul1pqcmur01) and it success
- Upgrade the cluster sw
- Run climgcheck on node2(kul1pqcmur02) and it failed with below errors
Clmigchek ran successfully on node1 and upgrade the cluster filesets
Climigcheck throws the below error on the node2 and unable to proceed the upgrade
PowerHA System Mirror migration check program.
Run date: Tue Mar 31 16:05:25 MYT 2015 on machine kul1pqcmur02
ck_user: Called
ck_version: Called
ck_version: Version check is OK
write_rhosts: Called
write_rhosts: ERROR - Unable to get hostname from node kul1pqcmur01 using cl_rsh
Error termination on: Tue Mar 31 16:05:25 MYT 2015
Action taken:
- Ensure the rhosts file has the proper entry on both nodes
- Ensure the rhosts permission has set to 600
Due to time constraint, have rollback powerHA to 6.1.0.9 but i wonder how the rsh work in 6.1.0.9 with the same rhosts entry.
Did anyone of you faced this kind of issue?
Appreciate your help and reply.
Cheers:b:
Try running the script on the second node with verbose logging at maximum to see if it reveals more information about the failure. Also, do the regular checks, ping the host by name and by ip
export VERBOSE_LOGGING=high
clmigcheck
Couldn't find what is the problem
# export VERBOSE_LOGGING=high
# clmigcheck
clmigcheck[4011]: /usr/es/sbin/cluster/utilities/cl_get_path -S
clmigcheck[4011]: 2> /dev/null
clmigcheck[4011]: OP_SEP='~'
clmigcheck[4012]: OP_SEP='~'
clmigcheck[4013]: MPING_PARAMS4='-v -c 5 -a 228.168.101.43'
clmigcheck[4014]: MPING_PARAMS6='-v -c 5 -6'
clmigcheck[4015]: MPING_PARAMS='-v -c 5 -a 228.168.101.43'
clmigcheck[4021]: trap cleanup 0 11
clmigcheck[4022]: trap user_exit 1 2 15
clmigcheck[4028]: ERR=/tmp/5308538.err
clmigcheck[4034]: [ -f /tmp/clmigcheck ]
clmigcheck[4039]: [[ -d /tmp/clmigcheck ]]
clmigcheck[4041]: [ -f /tmp/clmigcheck/clmigcheck.log ]
clmigcheck[4043]: print 'Saving existing /tmp/clmigcheck/clmigcheck.log to /tmp/clmigcheck/clmigcheck.log.bak'
Saving existing /tmp/clmigcheck/clmigcheck.log to /tmp/clmigcheck/clmigcheck.log.bak
clmigcheck[4044]: mv -f /tmp/clmigcheck/clmigcheck.log /tmp/clmigcheck/clmigcheck.log.bak
clmigcheck[4044]: 2> /dev/null
clmigcheck[4062]: date
clmigcheck[4062]: hostname
clmigcheck[4062]: echo 'PowerHA System Mirror migration check program.\nRun date: Wed Apr 15 15:53:29 MYT 2015 on machine kul1pqcmur02\n'
clmigcheck[4062]: 1> /tmp/clmigcheck/clmigcheck.log
clmigcheck[4071]: [ '' = -h ]
clmigcheck[4071]: [ '' = '-?' ]
clmigcheck[4080]: [ 0 != 0 ]
clmigcheck[4101]: echo
clmigcheck[4101]: sed 's/\-//g'
clmigcheck[4101]: cmdline_flags=''
clmigcheck[4107]: ck_user
ck_user[971]: log 'ck_user: Called\n'
log[896]: echo ck_user: 'Called\n'
log[896]: 2> /dev/null log[897]: return 0
ck_user[973]: whoami
ck_user[973]: [ root != root ]
clmigcheck[4108]: ck_version
ck_version[1010]: log 'ck_version: Called\n'
log[896]: echo ck_version: 'Called\n'
log[896]: 2> /dev/null log[897]: return 0
ck_version[1011]: rm -f /tmp/5308538.err
ck_version[1011]: 1> /dev/null 2>& 1
ck_version[1014]: lslpp -lcqOr cluster.es.server.rte
ck_version[1014]: 2> /dev/null
ck_version[1014]: cut -f3 -d:
ck_version[1014]: read version release mod fix
ck_version[1014]: IFS=.
ck_version[1021]: (( 6 <= 6 ))
ck_version[1023]: log 'ck_version: Version check is OK\n'
log[896]: echo ck_version: Version check is 'OK\n'
log[896]: 2> /dev/null log[897]: return 0
ck_version[1031]: return 0
clmigcheck[4114]: write_rhosts
write_rhosts[1172]: log 'write_rhosts: Called\n'
log[896]: echo write_rhosts: 'Called\n'
log[896]: 2> /dev/null log[897]: return 0
write_rhosts[1177]: 1> /tmp/rhosts
write_rhosts[1182]: odmget -q 'object = VERBOSE_LOGGING' HACMPnode
write_rhosts[1182]: ODMDIR=/etc/objrepos
write_rhosts[1183]: grep name
write_rhosts[1184]: cut -d '"' -f2
write_rhosts[1182]: nlist=$'kul1pqcmur01\nkul1pqcmur02'
write_rhosts[1188]: /usr/es/sbin/cluster/utilities/cl_rsh -n kul1pqcmur01 hostname
rshexec: cannot connect to node kul1pqcmur01
write_rhosts[1188]: hname=''
write_rhosts[1190]: (( 1 == 0 ))
write_rhosts[1196]: log 'write_rhosts: ERROR - Unable to get hostname from node kul1pqcmur01 using cl_rsh\n'
log[896]: echo write_rhosts: ERROR - Unable to get hostname from node kul1pqcmur01 using 'cl_rsh\n'
log[896]: 2> /dev/null log[897]: return 0
write_rhosts[1197]: error $'Unable to get hostname from node kul1pqcmur01\nusing cl_rsh\\n'
error[529]: syntax=$'Unable to get hostname from node kul1pqcmur01\nusing cl_rsh\\n'
error[529]: typeset syntax
error[531]: [[ $'Unable to get hostname from node kul1pqcmur01\nusing cl_rsh\\n' == no ]]
error[531]: [[ $'Unable to get hostname from node kul1pqcmur01\nusing cl_rsh\\n' == yes ]]
error[536]: print $'\\nERROR: Unable to get hostname from node kul1pqcmur01\nusing cl_rsh\\n'
error[536]: 1>& 2
ERROR: Unable to get hostname from node kul1pqcmur01
using cl_rsh
error[539]: [[ -s /tmp/5308538.err ]]
error[548]: print
error[548]: 1>& 2
error[550]: [[ $'Unable to get hostname from node kul1pqcmur01\nusing cl_rsh\\n' == yes ]]
error[553]: date
error[553]: log 'Error termination on: Wed Apr 15 15:53:29 MYT 2015 \n'
log[896]: echo Error termination on: Wed Apr 15 15:53:29 MYT 2015 '\n'
log[896]: 2> /dev/null log[897]: return 0
error[554]: exit 1
clmigcheck[1]: cleanup
This is the problem:
Check /etc/cluster/rhosts on both sides and restart clcomd
Both the node having the below entry
cat /etc/cluster/rhosts
kul1pqcmur01
kul1pqcmur02
Restarted the clcomd in kul1pqcmur01 node & restarted clcomd & clcomdES in kul1pqcmur02, tried to run the date command and it fails with the same error
there is no magic in clcomd. If cl_rsh fails, your /etc/cluster/rhosts configuration is wrong. It may be a problem with your /etc/hosts and/or DNS resolution. Try to enter all possible host names and IP addresses of both your hosts into rhosts.