Solaris 10 system boots into Maintenance mode

brownwrap · May 31, 2012, 6:21pm

I have seen similar threads on this issue, but I have not seen a fix. Basically I am getting a lot of rcp bind errors. Below find the output of "svcs -xv". not sure where to begin:

# svcs -xv
svc:/network/rpc/bind:default (RPC bindings)
 State: maintenance since Fri May 25 14:13:18 2012
Reason: Start method failed repeatedly, last exited with status 10.
   See: http://sun.com/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 1M rpcbind
   See: /var/svc/log/network-rpc-bind:default.log
Impact: 12 dependent services are not running:
        svc:/network/rpc/gss:default
        svc:/network/nfs/client:default
        svc:/system/filesystem/autofs:default
        svc:/network/nfs/server:default
        svc:/network/rpc/rstat:default
        svc:/network/nfs/nlockmgr:default
        svc:/network/nfs/status:default
        svc:/network/nfs/cbd:default
        svc:/network/nfs/mapid:default
        svc:/network/rpc/smserver:default
        svc:/network/rpc/rusers:default
        svc:/network/nfs/rquota:default

svc:/network/rpc-100235_1/rpc_ticotsord:default (100235)
 State: maintenance since Thu May 31 14:53:45 2012
Reason: Restarter svc:/network/inetd:default gave no explanation.
   See: http://sun.com/msg/SMF-8000-9C
Impact: This service is not running.

svc:/application/stosreg:default (Service Tag OS Registry Inserter)
 State: maintenance since Fri May 25 13:42:32 2012
Reason: Start method failed repeatedly, last exited with status 10.
   See: http://sun.com/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 1M stclient
   See: /var/svc/log/application-stosreg:default.log
Impact: This service is not running.

jlliagre · June 1, 2012, 1:32am

Start by looking at this log file:

tail  /var/svc/log/network-rpc-bind:default.log

incredible · June 1, 2012, 3:57am

u should check all ur File Systems are available or not.
go into each mount point and check data is available or not.
check /etc/vfstab entries with df -h output.
i think its problem with mounting the FS at boot time.
comment the vfstab entries. export any ZFS mounts (if any)
try, svcadm clear filesystem/local:default and see if it comes up to multiuser first

jlliagre · June 1, 2012, 4:09am

Why ? svcs doesn't report any file filesystem/local:default issues.

incredible · June 1, 2012, 4:22am

hhhmm.. makes sense.. but not sure if the svcs -xv has more to it or truncated...

brownwrap · June 1, 2012, 3:51pm

svcs -xv is NOT truncated, that is the entire output.

jlliagre · June 1, 2012, 6:07pm

Okay, so why don't you post the rpc-bind log file last lines ?

brownwrap · June 1, 2012, 6:14pm

code:

cat  /var/svc/log/network-rpc-bind:default.log
[ Dec 19 10:47:34 Rereading configuration. ]
[ Dec 19 10:48:30 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ Dec 19 10:48:31 Method "start" exited with status 0 ]
[ Dec 19 16:09:01 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ Dec 19 16:09:02 Method "start" exited with status 0 ]
[ Dec 20 07:10:28 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ Dec 20 07:10:28 Method "start" exited with status 0 ]
[ Feb 14 09:51:47 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ Feb 14 09:51:47 Method "start" exited with status 0 ]
[ Feb 23 13:32:21 Stopping because service disabled. ]
[ Feb 23 13:32:21 Executing stop method ("/lib/svc/method/rpc-bind stop 44") ]
[ Feb 23 13:32:26 Method "stop" exited with status 0 ]
[ Feb 23 13:39:07 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ Feb 23 13:39:08 Method "start" exited with status 0 ]
[ Mar 13 07:49:18 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ Mar 13 07:49:18 Method "start" exited with status 0 ]
[ May  8 10:34:52 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May  8 10:34:52 Method "start" exited with status 0 ]
[ May 10 15:51:56 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 10 15:51:56 Method "start" exited with status 10 ]
[ May 10 15:51:56 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 10 15:51:56 Method "start" exited with status 10 ]
[ May 10 15:51:57 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 10 15:51:57 Method "start" exited with status 10 ]
[ May 18 08:51:24 Leaving maintenance because disable requested. ]
[ May 18 08:51:24 Disabled. ]
[ May 18 09:34:07 Enabled. ]
[ May 18 09:34:16 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 18 09:34:16 Method "start" exited with status 10 ]
[ May 18 09:34:16 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 18 09:34:16 Method "start" exited with status 10 ]
[ May 18 09:34:16 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 18 09:34:16 Method "start" exited with status 10 ]
[ May 20 11:43:22 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 20 11:43:22 Method "start" exited with status 10 ]
[ May 20 11:43:22 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 20 11:43:22 Method "start" exited with status 10 ]
[ May 20 11:43:22 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 20 11:43:22 Method "start" exited with status 10 ]
[ May 25 13:42:28 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 25 13:42:28 Method "start" exited with status 10 ]
[ May 25 13:42:28 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 25 13:42:28 Method "start" exited with status 10 ]
[ May 25 13:42:28 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 25 13:42:28 Method "start" exited with status 10 ]
[ May 25 14:05:03 Leaving maintenance because disable requested. ]
[ May 25 14:05:03 Disabled. ]
[ May 25 14:05:17 Enabled. ]
[ May 25 14:05:17 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 25 14:05:17 Method "start" exited with status 10 ]
[ May 25 14:13:18 Leaving maintenance because clear requested. ]
[ May 25 14:13:18 Enabled. ]
[ May 25 14:13:18 Executing start method ("/lib/svc/method/rpc-bind start") ]
[ May 25 14:13:18 Method "start" exited with status 10 ]
[ May 31 15:50:36 Rereading configuration. ]
[ May 31 15:50:48 Rereading configuration. ]

DukeNuke2 · June 1, 2012, 8:03pm

so what happend on/shortly before May 10 15:51:56? did you install patches or added new software?

cjashu · June 4, 2012, 2:05am

Have you resolved the issue?

brownwrap · June 4, 2012, 8:16am

There are several people who work on this machine and I need to check with them to see what was done before this started happening.

cjashu · June 4, 2012, 8:23am

Seems like you are still having the same issue? I had a problem like that while studying. First I will ask you this:

While in maintenance mode, can you cat or vi into /etc/vfstab file?
We will proceed from there.

brownwrap · June 4, 2012, 9:09am

I am at home now, the se3rver is at work.

cjashu · June 4, 2012, 12:30pm

Ok your server can't mount root file system.

You have to boot the OS from cdrom then fix the problem in etcvfstab. You will be able to boot the server but might still go to maintenance mode. Not to worry, all you have to do is restore the service repository from its backup.

So fix the root file system first by booting from cdrom. Bring system ok prompt

 # init 0 
ok boot cdrom -s 
#fsck /dev/rdsk/c0t0d0s0 
#mount /dev/dsk/c0t0d0s0 /a
#TERM=sun
#export TERM
#vi /a/etc/vfstab
read this file carefully and fix the root file system entry. Save and quit.
#cd /
#umount /a 
# init 6 .

After this, the server may be fix or it might still come to maintenance mode. If it does, let us know so we can do the next step.

---------- Post updated at 12:30 PM ---------- Previous update was at 12:28 PM ----------

If you system mounts the root drive,
run the command below and follow instructions. Try using the seed repository

# /lib/svc/bin/restore_repository

brownwrap · June 4, 2012, 12:36pm

No, what I said, or meant to say is the root filesystem does mount. I see nothing wrong with /etc/vfstab.