File System Error: BAD SUPERBLOCK AT BLOCK 16: NUMBER OF DIRECTORIES OUT OF RANGE

Hi All,
we are having a file system error in one of our servers. The server failed to boot in usual user mode. Instead boot with single user mode and requesting to run a FSCK manually to repair the corrupted. see the below output.

Netra T2000, No Keyboard
Copyright 2008 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.28.9, 8064 MB memory available, Serial #9dddddddae.
.
 
 
 
 
 
 
Boot device: /pci@780/pci@0/pci@9/scsi@0/disk@1,0:a  File and args:
SunOS Release 5.10 Version Generic_142900-03 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: SVR
The / file system (/dev/md/rdsk/d0) is being checked.
 
 
WARNING - Unable to repair the / filesystem. Run fsck
manually (fsck -F ufs /dev/md/rdsk/d0).
 
 
May 12 17:21:42 svc.startd[7]: svc:/system/filesystem/usr:default: Method "/lib/svc/method/fs-usr" failed with exit status 95.
May 12 17:21:42 svc.startd[7]: system/filesystem/usr:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run
 
 
Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode
 
 
May 12 17:21:47 su: 'su root' succeeded for root on /dev/console
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
You have new mail.
Sourcing //.profile-EIS.....
root@SVR #
root@SVR #
root@SVR # fsck -F ufs /dev/md/rdsk/d0
** /dev/md/rdsk/d0
BAD SUPERBLOCK AT BLOCK 16: NUMBER OF DIRECTORIES OUT OF RANGE
 
 
LOOK FOR ALTERNATE SUPERBLOCKS WITH MKFS?
 
 
 
 
LOOK FOR ALTERNATE SUPERBLOCKS WITH NEWFS? Y
 
 
FOUND ALTERNATE SUPERBLOCK 32 WITH NEWFS
 
 
USE ALTERNATE SUPERBLOCK? y
 
 
FOUND ALTERNATE SUPERBLOCK AT 32 USING NEWFS
If filesystem was created with manually-specified geometry, using
auto-discovered superblock may result in irrecoverable damage to
filesystem and user data.
 
 
CANCEL FILESYSTEM CHECK? y
 
 
Please verify that the indicated block contains a proper
superblock for the filesystem (see fsdb(1M)).
root@SVR #
 

When we try to run the fsck it prompts to look for the superblocks using either MKFS or NEWFS, According to the oracle documentation, we need to select it correctly depending on the system. But we do not know from which command the file system is created. so with the warning given, we want get an advise on the best possible way to proceed for the file system repair.

Kindly advise on best possible way to proceed.

Thank You
Buddhike G

You're sounding nervous. Do you have a backup of this filesystem?

You can usually tell where the superblock copies are by using:
# newfs -N <filesystem device>

Then run fsck using an alternative superblock:
# fsck -F ufs -o b=<block number> <filesystem device>

Usually the second superblock copy is at block 32.

Thank You for your reply

There are no recent backups of the system.

here is my newfs output

root@SVR # newfs -N /dev/md/rdsk/d0
Warning: 4224 sector(s) in last cylinder unallocated
/dev/md/rdsk/d0:        209727360 sectors in 34136 cylinders of 48 tracks, 128 sectors
        102405.9MB in 2134 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
Initializing cylinder groups:
..........................................
super-block backups for last 10 cylinder groups at:
 208799264, 208897696, 208996128, 209094560, 209192992, 209291424, 209389856,
 209488288, 209586720, 209685152
root@SVR # 

Among the avilable super-block backups which one will be the best pick ?

Regards,
Buddhike G

I'd always go for 32 first. DON'T use -y switch on fsck and just see how many questions it asks you. If too many eg. 30+ , then abort.

---------- Post updated 14-05-15 at 12:01 PM ---------- Previous update was 13-05-15 at 04:04 PM ----------

You could just put the -n switch on the fsck to prevent it modifying anything. It will then just list how much damage it thinks there is but won't try to fix anything.

Hi hicksd8,

Good to know about the -n switch, So I will run with -n and see it first.

Regards,
Buddhike G

Hi All,

None of the recovery methods work for my issue. So we have done a fresh installation of the system

Thank You all for your suggestions

Regards,
Buddhike G