Failed to mount /usr

I recently ran the Solaris 10 upgrade 10 and everything went fine. About 2 months later after rebooting and getting ready to load the latest upgrade that came out, I am getting this error.

ERROR: svc:/system/filesytem/root: default failed to mount /usr (see 'svc -x' for details)

svc.startd[7]: scv: /system/filesystem/root:default:Method "/lib/svc/method/fs-root" failed with exit status 95
[system/filesystem/root:default failed fatally (see 'svc -x for details)]

Request system Maintenance mode
(see /lib/svc/share/README for more information)
Console login service(s) cannot run.

I have booted with the solaris cd into single user mode, mount s0 to /mnt, checked the vfstab and made sure everything is in order. I have ran fsck on all partitions and pretty much everything else that I have found on a google search.

Anyone have any ideas as to what is going on?

Have you tried sunning "svc -x" as suggested. In most of the cases, it will actually tell you the command to run for the remedy. Is /usr a separate filesystem? Does it have a SAN back-end or it's a local disk? Is the LUN available? etc. etc. These questions come into mind for this sort of troubleshooting.

I ran the svcs -x command and there earlier this morning and did not receive any output from it. /usr is mounted on a local disk

c1t0d0:
s0 /
s1 swap
s3 /usr
s4 /var
s5 /opt
s6 /var/audit
s7 /home

Have you checked /var/adm/messages for possible error messages? What happens when you try to manually mount /usr?

I checked /var/adm/messages and there was nothing in there. I have mounted /usr and there are not any directories or files underneath that as well.

Try to fsck all the filesystems multiple times, sounds crazy but sometimes just one time is not enough. Then mount all the filesystems and try to check files and directory contents. if you have similar installation on another system you can try to count the files and sizes and see if they differ a lot with the problematic one. Check the content of the most important files in /etc directory like system, passwd, shadow, vfstab, svc/repository.db, hosts, inittab.. etc.
If you still have problems take the disk out and try to boot on another HW... if it works then the problem is on the other cabinet.
If it doesn't boot restore from backup.
You can also try to boot with boot -m debug, or boot -v
Hope it helps... there are many other things you can try but that's a beginning

Juan

This could be caused by a number of problems.

  1. Is the /usr filesystem there at all? If you boot single user (boot cdrom -s) and mount the /usr filesystem manually (eg. under /a), what can you then see? Can you fsck the filesystem? (unmounted of course) Does it look right?

  2. Perhaps the root filesystem driver configuration is screwed. Try telling the system to reconfigure on boot by using the -r switch (boot -r). What happens?

  3. Perhaps the software cannot address the hardware due to a device screw up on the root filesystem. eg. /dev/dsk/c?t?d?s? do not allow the O/S to talk to the hardware. If so, you can tell it to rebuild the whole lot.

Boot single user (boot cd -s) USING THE MEDIA FOR THE O/S VERSION YOU ARE NOW RUNNING and mount your harddisk root filesystem under /a, then delete and reconstruct your disk devices:

 
rm -f /a/dev/rdsk/c*
rm -f /a/dev/dsk/c*
rm -f /a/dev/cfg/c*
devfsadm -r /a -p /a/etc/path_to_inst

Now check that (/a)/etc/vfstab looks right. Take your time. Use format command to view device names and partition tables (slices) of your drives (especially the /usr one giving you trouble). When you are sure vfstab is right, shutdown the system.

Now be sure to boot with the -r switch (boot -r) the first time you boot and the system will reconfigure all your disk devices and should mount /usr provided you have given the correct information to Solaris in vfstab.