syslog not coming up

Hi All,

I have a whole root zone, tsesbd02-zesbx01. On this zone, syslog service is not coming up. While it is working good on Global zone. Below are commands from tsesbd02-zesbx01

root@tsesbd02-zesbx01:/# svcs -a svc:/system/system-log:default
svcs: -a ignored when used with arguments.
STATE          STIME    FMRI
maintenance    12:35:29 svc:/system/system-log:default
root@tsesbd02-zesbx01:/# svcs -xv svc:/system/system-log:default
svc:/system/system-log:default (system log)
 State: maintenance since Wed Mar 14 12:35:29 2012
Reason: Start method exited with $SMF_EXIT_ERR_CONFIG.
   See: http://sun.com/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 1M syslogd
   See: /var/svc/log/system-system-log:default.log
Impact: This service is not running.
root@tsesbd02-zesbx01:/# tail -30 /var/svc/log/system-system-log:default.log
[ Mar 14 12:13:45 Executing stop method (:kill) ]
[ Mar 14 12:13:45 Executing start method ("/lib/svc/method/system-log") ]
svccfg: Unexpected libscf error on line 11040 of svccfg_libscf.c: server has insufficient resources.
svccfg: Property group already exists.
svccfg: Unexpected libscf error on line 11704 of svccfg_libscf.c: server has insufficient resources.
Failed to set remote syslog-logging
[ Mar 14 12:13:46 Method "start" exited with status 96 ]
[ Mar 14 12:20:02 Leaving maintenance because clear requested. ]
[ Mar 14 12:20:02 Enabled. ]
[ Mar 14 12:20:02 Executing start method ("/lib/svc/method/system-log") ]
svccfg: Unexpected libscf error on line 11040 of svccfg_libscf.c: server has insufficient resources.
svccfg: Property group already exists.
svccfg: Unexpected libscf error on line 11704 of svccfg_libscf.c: server has insufficient resources.
Failed to set remote syslog-logging
[ Mar 14 12:20:03 Method "start" exited with status 96 ]
[ Mar 14 12:27:19 Leaving maintenance because clear requested. ]
[ Mar 14 12:27:19 Enabled. ]
[ Mar 14 12:27:19 Executing start method ("/lib/svc/method/system-log") ]
svccfg: Unexpected libscf error on line 11040 of svccfg_libscf.c: backend is read-only.
svccfg: Unexpected libscf error on line 10816 of svccfg_libscf.c: backend is read-only.
svccfg: Unexpected libscf error on line 11704 of svccfg_libscf.c: backend is read-only.
Failed to set remote syslog-logging
[ Mar 14 12:35:28 Leaving maintenance because clear requested. ]
[ Mar 14 12:35:28 Enabled. ]
[ Mar 14 12:35:28 Executing start method ("/lib/svc/method/system-log") ]
svccfg: Unexpected libscf error on line 11040 of svccfg_libscf.c: backend is read-only.
svccfg: Unexpected libscf error on line 10816 of svccfg_libscf.c: backend is read-only.
svccfg: Unexpected libscf error on line 11704 of svccfg_libscf.c: backend is read-only.
Failed to set remote syslog-logging
[ Mar 14 12:35:29 Method "start" exited with status 96 ]
root@tsesbd02-zesbx01:/#

Please help, how to clear this maintenance flag and activate it

Regards

Can you post contents of /etc/syslog.conf ?

root@tsesbd02-zesbx01:/# cat /etc/syslog.conf
#
# Copyright (c) 2000-2002 by Sun Microsystems, Inc.
# All rights reserved.
#
#ident  "@(#)syslog.conf        2.3     02/02/21        SMI"
#
# This "syslog.conf" file was installed by JASS.  This
# file should be used to log information both locally as
# well as to a centralized log server (or servers) so that
# proactive log analysis can be done.
*.err;kern.notice;auth.notice   /dev/console
*.alert                         root
*.emerg                         *
*.debug                         /var/adm/messages
# *.debug                       @loghost1
# *.debug                       @loghost2
#user.debug          /var/mqsi/log/user.log
user.info                       /var/mqsi/log/user.log

I found similar issue here. Try killing svc.configd and then clearing syslog service.

Bartus, I already tried that and that didn't helped. It seems like his root cause was different

root@tsesbd02-zesbx01:/# ps -ef | grep -i svc.configd
    root 27101 21113   0 14:34:43 pts/2       0:00 grep -i svc.configd
    root  6614 18078   0 12:26:50 ?           0:15 /lib/svc/bin/svc.configd
root@tsesbd02-zesbx01:/# kill -9 6614
root@tsesbd02-zesbx01:/# ps -ef | grep -i svc.configd
    root 27103 18078   0 14:34:50 ?           0:01 /lib/svc/bin/svc.configd
    root 27106 21113   0 14:34:53 pts/2       0:00 grep -i svc.configd
root@tsesbd02-zesbx01:/# svcadm clear svc:/system/system-log:default
root@tsesbd02-zesbx01:/# svcs -a | grep -i /system/system-log
maintenance    14:35:17 svc:/system/system-log:default

Have you tried stopping the zone and booting it again?

Not yet. Few applications are running on zone, so I was trying to start syslog service without restarting zone. Does that log says something

/var/svc/log/system-system-log:default.log 

?

It should say why the service is not starting. Unfortunately I've never seen the error messages that you have there. It seems to me that the zone itself is broken in some way (not the syslog service). I might be wrong though.

Yes, you may be right. I am checking with Oracle on this. Will let you know, if I get something

Have you done any work on the whole root zone, all my zones are sparse so I have no experience of whole root zones - but i have seen a very similar error generated during the live upgrade process.

What if anything was done while the zone was down or being rebooted?

Regards

Dave

No Dave, there was no recent maintenance on global as well as non global zone

root@tsesbd02-zesbx01:/# uptime
  2:43pm  up 287 day(s),  3:49,  1 user,  load average: 0.43, 0.41, 0.41
 
root@tsesbd02:/# uptime
  3:02pm  up 330 day(s), 21:51,  1 user,  load average: 0.48, 0.51, 0.52

Well I'll just stay with this thread - I'd like to know the outcome of this one as I've had a similar message in the past and clock change weekend is my maintenamce outage where I have several large servers to patch with a three figure number of sparse zones/containers.

Admittedly each has only a single application, but the owners of the apps are keen to point fingers.

I'll be here for a couple of hours.

Regards

Dave

As per Oracle Support, respository.db was corrupted on this zone, so I had to copy it from another similar zone. Now things are fine on this.
There may be other solution also like svccfg delete and svccfg import, but I am not sure

It should also be possible to recover repository from its backup on the same zone using /lib/svc/bin/restore_repository .