RSH intermittent error rshd: 0826-813 Permission is denied.

I am getting an error from one node in a set with RSH setup between them, node one will connect to node two every other time (consistently), however node to connects to node one every time without problem. Here is what I am seeing, makes no sense to me. Can anyone help?

sbhcprdb01<root>: rsh sbhcprdb02 date
Mon Jun 14 14:15:01 EDT 2010
sbhcprdb01<root>: rsh sbhcprdb02 date
rshd: 0826-813 Permission is denied.
sbhcprdb01<root>: rsh sbhcprdb02 date
Mon Jun 14 14:15:05 EDT 2010
sbhcprdb01<root>: rsh sbhcprdb02 date
rshd: 0826-813 Permission is denied.
sbhcprdb01<root>: rsh sbhcprdb02 date
Mon Jun 14 14:15:10 EDT 2010
sbhcprdb01<root>: rsh sbhcprdb02 date
rshd: 0826-813 Permission is denied.
sbhcprdb01<root>: rsh sbhcprdb02 date
Mon Jun 14 14:15:14 EDT 2010

sbhcprdb01<root>: rlogin sbhcprdb02
root's Password:
Connection closed.
sbhcprdb01<root>: rlogin sbhcprdb02
*******************************************************************************
*                                                                             *
*                                                                             *
*  Welcome to AIX Version 6.1!                                                *
*                                                                             *
*                                                                             *
*  Please see the README file in /usr/lpp/bos for information pertinent to    *
*  this release of the AIX Operating System.                                  *
*                                                                             *
*                                                                             *
*******************************************************************************
Last unsuccessful login: Mon Jun 14 11:03:18 EDT 2010 on /dev/pts/0 from bchcprdb01.bch.barbhosp.com
Last login: Mon Jun 14 13:19:18 EDT 2010 on /dev/pts/0 from sbhcprdb01.bch.barbhosp.com

sbhcprdb02<root>: exit

sbhcprdb01<root>: host sbhcprdb02
sbhcprdb02.bch.barbhosp.com is 10.2.6.5
sbhcprdb01<root>: host 10.2.6.5
sbhcprdb02.bch.barbhosp.com is 10.2.6.5

sbhcprdb01<root>: cat .rhosts
sbhcprdb02.bch.barbhosp.com  root
sbhcprdb02  root


sbhcprdb01<root>: cat /etc/hosts |grep sbhcprdb02
10.2.6.5        sbhcprdb02      sbhcpr02-02     BCH-PRD2        SBH-PRD2
192.168.89.2    sbhcprdb02-gige         sbhcpr02-02-gige
sbhcprdb01<root>:



sbhcprdb02<root>: rsh sbhcprdb01 date
Mon Jun 14 14:37:07 EDT 2010
sbhcprdb02<root>: rsh sbhcprdb01 date
Mon Jun 14 14:37:09 EDT 2010
sbhcprdb02<root>: rsh sbhcprdb01 date
Mon Jun 14 14:37:10 EDT 2010
sbhcprdb02<root>: rsh sbhcprdb01 date
Mon Jun 14 14:37:11 EDT 2010

sbhcprdb02<root>: host sbhcprdb01
sbhcprdb01.bch.barbhosp.com is 10.2.6.4
sbhcprdb02<root>: host 10.2.6.4
bchcprdb01.bch.barbhosp.com is 10.2.6.4,  Aliases:   sbhcprdb01.bch.barbhosp.com

sbhcprdb02<root>: cat /etc/hosts |grep sbhcprdb01
10.2.6.4        sbhcprdb01      sbhcpr01-02     BHC-PRD1        SBH-PRD1
192.168.89.1    sbhcprdb01-gige         sbhcpr01-02-gige

sbhcprdb02<root>: cat .rhosts
sbhcprdb01.bch.barbhosp.com root
sbhcprdb01 root
sbhcprdb02<root>:

Could you post the output of "netstat -rn" and "netstat -in" from both nodes.

Also, I did forget to mention that these two nodes are running in virtualized environments on two p6 550's, single vio and one node on each 550. I have checked the error logs on both nodes and VIO'S and not getting anything from that.

sbhcprdb01<root>: netstat -rn
Routing tables
Destination        Gateway           Flags   Refs     Use  If   Exp  Groups

Route Tree for Protocol Family 2 (Internet):
default            10.10.0.1         UG        1       830 en0      -      -
10.0.0.0           10.2.6.4          UHSb      0         0 en0      -      -   =>
10/8               10.2.6.4          U         5     15236 en0      -      -
10.2.6.4           127.0.0.1         UGHS      2        61 lo0      -      -
10.255.255.255     10.2.6.4          UHSb      0         4 en0      -      -
127/8              127.0.0.1         U        13      8530 lo0      -      -
192.168.89.0       192.168.89.1      UHSb      0         0 en1      -      -   =>
192.168.89/24      192.168.89.1      U         0     70949 en1      -      -
192.168.89.1       127.0.0.1         UGHS      0        85 lo0      -      -
192.168.89.255     192.168.89.1      UHSb      0         4 en1      -      -

Route Tree for Protocol Family 24 (Internet v6):
::1                ::1               UH        0        48 lo0      -      -
sbhcprdb01<root>: netstat -in
Name  Mtu   Network     Address           ZoneID    Ipkts Ierrs    Opkts Oerrs  Coll
en0   1500  link#2      86.7c.8b.64.8d.14      -   484295     0    16680     0     0
en0   1500  10          10.2.6.4               -   484295     0    16680     0     0
en1   1500  link#3      86.7c.8b.64.8d.1e      -   505400     0    70970     0     0
en1   1500  192.168.89  192.168.89.1           -   505400     0    70970     0     0
lo0   16896 link#1                             -     8851     0     8876     0     0
lo0   16896 127         127.0.0.1              -     8851     0     8876     0     0
lo0   16896 ::1                                0     8851     0     8876     0     0
sbhcprdb01<root>:


sbhcprdb02<root>: netstat -rn
Routing tables
Destination        Gateway           Flags   Refs     Use  If   Exp  Groups

Route Tree for Protocol Family 2 (Internet):
default            10.10.0.1         UG        1       870 en0      -      -
10.0.0.0           10.2.6.5          UHSb      0         0 en0      -      -   =>
10/8               10.2.6.5          U         6     10833 en0      -      -
10.2.6.5           127.0.0.1         UGHS      0        41 lo0      -      -
10.255.255.255     10.2.6.5          UHSb      0         4 en0      -      -
127/8              127.0.0.1         U        14      8528 lo0      -      -
192.168.89.0       192.168.89.2      UHSb      0         0 en1      -      -   =>
192.168.89/24      192.168.89.2      U         0      4227 en1      -      -
192.168.89.2       127.0.0.1         UGHS      0        55 lo0      -      -
192.168.89.255     192.168.89.2      UHSb      0         4 en1      -      -

Route Tree for Protocol Family 24 (Internet v6):
::1                ::1               UH        0        48 lo0      -      -
sbhcprdb02<root>: netstat -in
Name  Mtu   Network     Address           ZoneID    Ipkts Ierrs    Opkts Oerrs  Coll
en0   1500  link#2      de.8e.86.2d.8d.14      -   480814     0    12212     0     0
en0   1500  10          10.2.6.5               -   480814     0    12212     0     0
en1   1500  link#3      de.8e.86.2d.8d.1e      -   473957     0     4251     0     0
en1   1500  192.168.89  192.168.89.2           -   473957     0     4251     0     0
lo0   16896 link#1                             -     8752     0     8778     0     0
lo0   16896 127         127.0.0.1              -     8752     0     8778     0     0
lo0   16896 ::1                                0     8752     0     8778     0     0
sbhcprdb02<root>:

Had the same problem once - there was a wrong entry on one of our servers being used for DNS resolving (we have plenty of them helping out on that job). From 10 tries about 9 were working.
When the box with the wrong or missing entry was used (I think there was an alias missing on that DNS server), I got an error.
I noticed it while hitting repeatedly nslookup or host several times one after another like a maniac; it looked like in terms of:

ok
ok
ok
ok
ok
ok
ok
wrong
ok
ok

Also check maybe your /etc/netsvc.conf for resolving order and just in case /etc/hosts for wrong entries.

how long you search adress?

sbhcprdb01<root>: date; host sbhcprdb02 ; date

how much diferent your times:

sbhcprdb01<root>: date; rsh sbhcprdb02 date ; date

you can do it on sbhcprdb02

sbhcprdb02<root>: date; host sbhcprdb01 ; date
sbhcprdb02<root>: date; rsh sbhcprdb01 date ; date

I think it's good idea.

Hi

I just had a similar experience. It turned out that the A machine I was using had multiple names in the DNS server. I found this by running nslookup on the B machine. To fix it, I had to make a separate .rhosts entry line for each of the names returned by nslookup.

Yep, looks like node two is failing to identify node one via the DNS reverse name lookup in one of its DNS servers.