Svc messages flooding the system logs every second

notreallyhere · November 27, 2012, 4:17am

Hi all

I have a newly installed Oracle X2-4 server running Solaris 10 x86 with the latest patches.
I have one non-global zone configured running an Oracle DB instance.
After configuring IPMP failover between two NICs on the server and rebooting I am seeing the /var/adm/messages being flooded every second with the following output:

Nov 26 10:21:24 host_server svc.startd[9]: [ID 122153 daemon.warning] svc:/system/vxpbx:default: Method or service exit timed out.  Killing contract 7299.
Nov 26 10:21:24 host_server svc.startd[9]: [ID 122153 daemon.warning] svc:/application/management/wbem:default: Method or service exit timed out.  Killing contract 7301.
Nov 26 10:21:24 host_server svc.startd[9]: [ID 122153 daemon.warning] svc:/application/management/hwmgmtd:default: Method or service exit timed out.  Killing contract 95.
Nov 26 10:21:24 host_server svc.startd[9]: [ID 122153 daemon.warning] svc:/system/webconsole:console: Method or service exit timed out.  Killing contract 7306.
Nov 26 10:21:24 host_server svc.startd[9]: [ID 122153 daemon.warning] svc:/system/zones:default: Method or service exit timed out.  Killing contract 7300.

svcs -xv returns a blank output.
I've tried disabling those services but to no avail. I've also rebooted the server.
I am not seeing any negative impact in terms of server functionality. The IPMP failover works fine, the zone is running perfectly as well.

Any help in resolving this would be highly appreciated.

jim_mcnamara · November 27, 2012, 4:35am

Something is not configured correctly - obviously. And things are not fine.
You created (I hope) two

/etc/hostname.[NIC card name]
like:
/etc/hostname.bge0

files
Your actual names will be different.

Please show the output of:

cat /etc/hostname.ce0  # or whatever it is called
cat /etc/hostname.ce1  # or whatever it is called
grep solaris /etc/hostname
ifconfig -a
ipmpstat -g

Have you tried:

if_mpadm -d [name of primary NIC]

Don't do it yet, please. You may cause more problems.

notreallyhere · November 27, 2012, 5:45am

Hi Jim

This server was one of 7 that I was busy configuring with IPMP failover. I've used the same procedure as on the other servers and this is the only one that is giving me this issue.

Below are some outputs:

:~# grep solaris /etc/hostname
grep: can't open /etc/hostname

:~# dladm show-dev
igb0            link: up        speed: 1000  Mbps       duplex: full
igb1            link: up        speed: 1000  Mbps       duplex: full
igb2            link: unknown   speed: 0     Mbps       duplex: half
igb3            link: unknown   speed: 0     Mbps       duplex: half

:~# ls -l /etc/hostname.*
-rw-r--r--   1 root     root         117 Nov 25 12:04 /etc/hostname.igb0
-rw-r--r--   1 root     root          76 Nov 25 12:03 /etc/hostname.igb1

:~# cat /etc/hostname.igb0
host_server-igb0 group ipmp_prod netmask + broadcast + -failover deprecated up addif host_server  netmask + broadcast + up

:~# cat /etc/hostname.igb1
host_server-igb1 group ipmp_prod netmask + broadcast + -failover deprecated up

:~# cat /etc/hosts
xx.xx.xx.xx   host_server        host_server.domain       loghost
xx.xx.xx.xx   host_server-igb0
xx.xx.xx.xx   host_server-igb1

:~# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
lo0:1: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        zone db_zone
        inet 127.0.0.1 netmask ff000000
igb0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5
        inet xx.xx.xx.xx netmask fffffc00 broadcast xx.xx.xx.xx
        groupname ipmp_prod
        ether x:x:x:x:x
igb0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
        inet xx.xx.xx.xx netmask fffffc00 broadcast xx.xx.xx.xx
igb0:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
        zone db_zone
        inet xx.xx.xx.xx netmask fffffc00 broadcast xx.xx.xx.xx
igb1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
        inet xx.xx.xx.xx netmask fffffc00 broadcast xx.xx.xx.xx
        groupname ipmp_prod
        ether x:x:x:x:x

I have also run the if_mpadm command to test failover on both primary and secondary NICs (I do this by default to test IPMP after configuring it) and I have not lost my ssh session and failover works as it should.
Perhaps I have missed something and maybe you can spot it in some of the outputs I've posted.

jim_mcnamara · November 27, 2012, 5:56am

And you ran

if_mpadm -r

to restore the primary NIC. I'm assuming ipmpstat shows ok on both NICs.

I don't see anything. However, some of those services that are having problems like system/zones are important, and should not be dumping logs.

Maybe someone else can see something. There are several folks here with Solaris admin background.

bartus11 · November 27, 2012, 6:01am

Can you post output of:

svcs -xv svc:/system/vxpbx:default svc:/system/zones:default

notreallyhere · November 27, 2012, 6:04am

I forgot to mention that I don't have that ipmpstat command on my Solaris installation. I ran a find on it now as well but came up with no results.

---------- Post updated at 01:04 PM ---------- Previous update was at 01:02 PM ----------

:~# svcs -xv svc:/system/vxpbx:default svc:/system/zones:default
svc:/system/vxpbx:default (Symantec Private Branch Exchange Service.)
 State: online since Sun Nov 25 11:51:02 2012
   See: /var/svc/log/system-vxpbx:default.log
Impact: None.

svc:/system/zones:default (Zones autoboot and graceful shutdown)
 State: online since Sun Nov 25 11:51:02 2012
   See: man -M /usr/share/man -s 5 zones
   See: man -M /usr/share/man -s 1M zonecfg
   See: /var/svc/log/system-zones:default.log
Impact: None.

bartus11 · November 27, 2012, 6:11am

And:

tail -20 /var/svc/log/system-vxpbx:default.log
tail -20 /var/svc/log/system-zones:default.log

notreallyhere · November 27, 2012, 6:29am

:~# tail -20 /var/svc/log/system-vxpbx:default.log
[ Nov 26 13:29:07 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:08 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:09 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:10 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:11 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:12 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:13 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:14 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:15 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:16 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:17 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:18 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:19 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:20 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:21 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:22 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:23 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:24 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:25 Method or service exit timed out.  Killing contract 7299 ]
[ Nov 26 13:29:26 Method or service exit timed out.  Killing contract 7299 ]

:~# tail -20 /var/svc/log/system-zones:default.log
[ Nov 26 13:29:19 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:20 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:21 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:22 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:23 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:24 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:25 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:26 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:27 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:28 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:29 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:30 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:31 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:32 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:33 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:34 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:35 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:36 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:37 Method or service exit timed out.  Killing contract 7300 ]
[ Nov 26 13:29:38 Method or service exit timed out.  Killing contract 7300 ]