Serious problem to fail boot up

My current OS using the solaris 9 and find the reboot , it returns the message even i restore , and boot -s

Hardware watchdog enabled 
Configuring ATM interfaces: 
NOTICE: VxVM not started 
configuring IPv4 interfaces: bge0 bge3 ce0 ce3. 
Hostname: devuardms01 
mount: mount-point /usr does not exist. 
/sbin/rcS: /usr/sbin/devfsadm: not found 
/sbin/rcS: /usr/bin/loadkeys: not found 
/etc/rcS.d/S35svm.init: rm: not found 
/etc/rcS.d/S35svm.init: grep: not found 
/etc/rcS.d/S35vxvm-startup1: /usr/lib/vxvm/voladm.d/lib/vxadm_lib.sh: not found 
/sbin/swapadd: expr: not found 
/sbin/swapadd: swap: not found 

WARNING - /usr/sbin/fsck not found. Most likely the 
mount of /usr failed or the /usr filesystem is badly 
damaged. The system is being halted. Either reinstall 
the system or boot with the -b option in an attempt 
to recover. 

syncing file systems... done 
Program terminated

The fundamental issue here is:

This may be the only problem or there might be more extensive filesystem damage.

Do you have the Solaris 9 install media? If so,

ok> book cdrom -s

to boot into single user from DVD.

At the # prompt, you can then mount your hard disk root filesystem under /a to inspect it. Something like:

# mount /dev/dsk/c0t0d0s0 /a

(assuming your root filesystem is c0t0d0s0)

Hi

Here is my server vfstab

/proc   -       /proc   proc    -       no      - 
 /dev/dsk/c1t0d0s1       -       -       swap    -       no      - 
 /dev/dsk/c1t0d0s0       /dev/rdsk/c1t0d0s0      /       ufs     1       no      - 
 /dev/dsk/c1t0d0s6       /dev/rdsk/c1t0d0s6      /usr    ufs     1       no      - 
 /dev/dsk/c1t0d0s5       /dev/rdsk/c1t0d0s5      /var    ufs     1       no      - 
 swap    -       /tmp    tmpfs   -       yes     - 
 #

I mount to / /var /usr , and view all the content , but try to reboot again and again . It still come out this issue.

Hardware watchdog enabled 
 Configuring ATM interfaces: 
 NOTICE: VxVM not started 
 configuring IPv4 interfaces: bge0 bge3 ce0 ce3. 
 Hostname: devuardms01 
 mount: mount-point /usr does not exist. 
 /sbin/rcS: /usr/sbin/devfsadm: not found 
 /sbin/rcS: /usr/bin/loadkeys: not found 
 /etc/rcS.d/S35svm.init: rm: not found 
 /etc/rcS.d/S35svm.init: grep: not found 
 /etc/rcS.d/S35vxvm-startup1: /usr/lib/vxvm/voladm.d/lib/vxadm_lib.sh: not found 
 /sbin/swapadd: expr: not found 
 /sbin/swapadd: swap: not found

I boot from solaris 9 or 10 to copy those missing file but still not to work to boot up successfully.

:confused:

---------- Post updated at 10:45 PM ---------- Previous update was at 10:40 PM ----------

Does the mountpoint /usr exist on your hard disk root filesystem?????

If not, then the booting system will not be able to mount the /usr filesystem and that will cause a big problem.

Boot from DVD:

ok> boot cdrom -s

When at # prompt:

# mount /dev/dsk/c1t0d0s0 /a
# cd /a
# ls -l

Does the mountpoint directory /usr exist????? Your error message says that it doesn't!! Post the output of ls -l

Copying files from a Solaris 10 media to a failing Solaris 9 installation will make things significantly worse...

I double check that i boot from Sol 9 CD but the kernel is latest one .

---------- Post updated at 11:14 AM ---------- Previous update was at 10:18 AM ----------

/dev/dsk/c1t0d0s0      9.8G   6.6G   3.1G    68%    /mnt
# ls -l
total 1894
drwxr-xr-x   2 root     root         512 Sep 24  2006 TT_DB
-rw-r--r--   1 root     other        505 Feb 22  2007 app.txt
drwxr-xr-x   2 root     other        512 Aug 22  2007 backup
-rwxr-xr-x   1 root     other        159 Nov 18  2011 backup.sh
lrwxrwxrwx   1 root     root           9 Apr 12 21:09 bin -> ./usr/bin
drwxr-xr-x   3 root     root         512 Dec 14  2006 cdrom
-rw-------   1 root     root      868392 Dec 18  2006 core
-rw-r--r--   1 root     other       1024 Dec 10  2006 cs_sp3.tar
drwxr-xr-x   2 root     other        512 Oct 26  2006 dctm
-rw-------   1 root     other         72 May  3  2016 dead.letter
drwxr-xr-x  17 root     sys         5120 Apr 11 18:56 dev
drwxr-xr-x   7 root     sys          512 Jul 19  2006 devices
drwxr-xr-x  64 root     sys         5120 Apr 15 20:55 etc
drwxr-xr-x   3 root     sys          512 Sep 26  2006 export
drwxr-xr-x  10 root     sys          512 Apr 11 02:37 kernel
lrwxrwxrwx   1 root     root           9 Apr 12 21:09 lib -> ./usr/lib
drwxr-xr-x   2 root     other        512 Mar 18 22:03 loaddgvol01
drwx------   2 root     root        8192 Jul 19  2006 lost+found
-rw-------   1 root     other      42510 Mar  8  2007 mbox
drwx------   2 root     other        512 Nov 27  2006 nsmail
drwxr-xr-x  46 root     sys         1024 Apr  5 23:50 opt
drwxr-xr-x  47 root     sys         2048 Apr 11 02:20 platform
drwxr-xr-x   2 root     sys         1536 Apr 11 02:59 sbin
drwxrwxrwx   4 root     other        512 Dec 10  2006 tempfs
-rw-r--r--   1 root     other        194 Apr 29  2014 tmpfile2
drwxrwxr-x   3 20000    20001        512 Oct 25  2006 u01
-rw-------   1 root     other          0 Feb 26 14:06 vm

I'm not sure what you mean by saying "i boot from Sol 9 CD but the kernel is latest one"???

You cannot run a Solaris 10 or 11 kernel on a system with Solaris 9 libraries, and there is no reason to believe that you'll be successful running a Solaris 9 kernel on a system with Solaris 10 or 11 libraries.

You should run a kernel on a system with libraries that have the same release numbers (major and minor and micro).

I confirm I boot from the Solaris 9 , which is the same version of OS.

Your file listing in post#6 shows that the /usr mountpoint is indeed missing.

So let's recreate it.

Boot from DVD:

ok> boot cdrom -s

When at # prompt:

# mount /dev/dsk/c1t0d0s0 /a
# cd /a
# mkdir usr

You can try rebooting now provided that was the only problem. Otherwise you should run fsck on the root filesystem to see what damage there is.

# umount /a
# fsck -n /dev/rdsk/c1t0d0s0

Note: Use the -n switch to ensure no modifiations to the filesystem initially until we know the extent of any damage.

# fsck -n /dev/rdsk/c1t0d0s0
** /dev/rdsk/c1t0d0s0 (NO WRITE)
** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cylinder Groups

CORRECT BAD CG SUMMARIES?  no

FRAG BITMAP WRONG
FIX?  no

FRAG BITMAP WRONG
FIX?  no

FRAG BITMAP WRONG
FIX?  no

FRAG BITMAP WRONG
FIX?  no

42084 files, 6897219 used, 3378913 free (1225 frags, 422211 blocks, 0.0% fragmen tation)

Now run that again without the -n switch and answer y to each question:

Boot from DVD:

ok> boot cdrom -s

When at # prompt:

# fsck /dev/rdsk/c1t0d0s0

and answer 'y' to each correction.

I assume that you have recreated the /usr mountpoint as per my revious post?