Unknown Disks "offline or reservation conflict"

MobileGSP · November 14, 2012, 4:39pm

Hi All,

I am a RH Linux admin that recently started working at a company with a number of SUN Servers so it's been an interesting transition.

Considering the last person left with very little documentation left behind so I have been picking up most issues half complete, or troubleshot.

After a recent poweroutage - planned so all servers were shut down healthy then powered back online network - storage - servers... (this is probably why my logs will show the date is Jan 2000 it looks like our battery is dead so the system clock reset).

So a user just came to me with mount points are not online, I did some quick checks and found that they were previously commented out of the /etc/vfstab which leads me to believe this is not a new problem, and whomever was last working on this server didn't completely fix it and just kept the server from having booting issues.

From some quick searches of the forum I have found a few commands that I can use to try and diagnose this, but I'm not sure where to go from here, I think my storage is gone... or failed.

The disks currently show as "drive type unknown" and there are logs in /var/adm/messages which indicate "offline or reservation conflict"

From everything I have read if i'm going to try and set the disk type I need to know exactly how it was configured previously to re-configure it in hopes of recovering any of the data that was previously on the array.

Below I put the output of any of the commands that look like they could help point me in the right direction, but I fear the worse considering these devices don't show up in most of the output.

Sun T2000

# uname -a
SunOS Servername 5.10 Generic_127127-11 sun4v sparc SUNW,SPARC-Enterprise-T2000

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@0,0
       1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@1,0
       2. c0t2d0 <drive type unknown>
          /pci@780/pci@0/pci@9/scsi@0/sd@2,0
       3. c0t3d0 <drive type unknown>
          /pci@780/pci@0/pci@9/scsi@0/sd@3,0
Specify disk (enter its number):2 

AVAILABLE DRIVE TYPES:
        0. Auto configure
        1. Quantum ProDrive 80S
        2. Quantum ProDrive 105S
        3. CDC Wren IV 94171-344
        4. SUN0104
        5. SUN0207
        6. SUN0327
        7. SUN0340
        8. SUN0424
        9. SUN0535
        10. SUN0669
        11. SUN1.0G
        12. SUN1.05
        13. SUN1.3G
        14. SUN2.1G
        15. SUN2.9G
        16. Zip 100
        17. Zip 250
        18. Peerless 10GB
        19. SUN72G
        20. other

# metadb
        flags           first blk       block count
     a m  p  luo        16              8192            /dev/dsk/c0t0d0s7
     a    p  luo        8208            8192            /dev/dsk/c0t0d0s7
     a    p  luo        16400           8192            /dev/dsk/c0t0d0s7
     a    p  luo        16              8192            /dev/dsk/c0t1d0s7
     a    p  luo        8208            8192            /dev/dsk/c0t1d0s7
     a    p  luo        16400           8192            /dev/dsk/c0t1d0s7

# metastat -p
d1 -m d11 d21 1
d11 1 1 c0t0d0s1
d21 1 1 c0t1d0s1
d0 -m d10 d20 1
d10 1 1 c0t0d0s0
d20 1 1 c0t1d0s0

# iostat -En

c0t2d0           Soft Errors: 0 Hard Errors: 77 Transport Errors: 0
Vendor: SEAGATE  Product: ST914602SSUN146G Revision: 0400 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 6 Device Not Ready: 70 No Device: 1 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c0t3d0           Soft Errors: 0 Hard Errors: 71 Transport Errors: 0
Vendor: SEAGATE  Product: ST914602SSUN146G Revision: 0400 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 6 Device Not Ready: 64 No Device: 1 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

# cat /var/adm/messages

Jan 17 11:18:35 ServerName scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0/sd@2,0 (sd7):
Jan 17 11:18:35 ServerName offline or reservation conflict
Jan 17 11:18:45 ServerName scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0/sd@2,0 (sd7):
Jan 17 11:18:45 ServerName offline or reservation conflict
Jan 17 11:19:05 ServerName scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0/sd@3,0 (sd8):
Jan 17 11:19:05 ServerName offline or reservation conflict
Jan 17 11:19:15 ServerName scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0/sd@3,0 (sd8):
Jan 17 11:19:15 ServerName offline or reservation conflict

# prtdiag
System Configuration:  Sun Microsystems  sun4v SPARC Enterprise T2000
System clock frequency: 200 MHz
Memory size: 16376 Megabytes

================================ Virtual CPUs ================================


CPU ID Frequency Implementation         Status
------ --------- ---------------------- -------
0      1200 MHz  SUNW,UltraSPARC-T1     on-line
1      1200 MHz  SUNW,UltraSPARC-T1     on-line
2      1200 MHz  SUNW,UltraSPARC-T1     on-line
3      1200 MHz  SUNW,UltraSPARC-T1     on-line
4      1200 MHz  SUNW,UltraSPARC-T1     on-line
5      1200 MHz  SUNW,UltraSPARC-T1     on-line
6      1200 MHz  SUNW,UltraSPARC-T1     on-line
7      1200 MHz  SUNW,UltraSPARC-T1     on-line
8      1200 MHz  SUNW,UltraSPARC-T1     on-line
9      1200 MHz  SUNW,UltraSPARC-T1     on-line
10     1200 MHz  SUNW,UltraSPARC-T1     on-line
11     1200 MHz  SUNW,UltraSPARC-T1     on-line
12     1200 MHz  SUNW,UltraSPARC-T1     on-line
13     1200 MHz  SUNW,UltraSPARC-T1     on-line
14     1200 MHz  SUNW,UltraSPARC-T1     on-line
15     1200 MHz  SUNW,UltraSPARC-T1     on-line
16     1200 MHz  SUNW,UltraSPARC-T1     on-line
17     1200 MHz  SUNW,UltraSPARC-T1     on-line
18     1200 MHz  SUNW,UltraSPARC-T1     on-line
19     1200 MHz  SUNW,UltraSPARC-T1     on-line
20     1200 MHz  SUNW,UltraSPARC-T1     on-line
21     1200 MHz  SUNW,UltraSPARC-T1     on-line
22     1200 MHz  SUNW,UltraSPARC-T1     on-line
23     1200 MHz  SUNW,UltraSPARC-T1     on-line
24     1200 MHz  SUNW,UltraSPARC-T1     on-line
25     1200 MHz  SUNW,UltraSPARC-T1     on-line
26     1200 MHz  SUNW,UltraSPARC-T1     on-line
27     1200 MHz  SUNW,UltraSPARC-T1     on-line
28     1200 MHz  SUNW,UltraSPARC-T1     on-line
29     1200 MHz  SUNW,UltraSPARC-T1     on-line
30     1200 MHz  SUNW,UltraSPARC-T1     on-line
31     1200 MHz  SUNW,UltraSPARC-T1     on-line

========================= IO Configuration =========================

            IO
Location    Type  Slot Path                                          Name                      Model
----------- ----- ---- --------------------------------------------- ------------------------- ---------
IOBD/NET0    PCIE IOBD                /pci@780/pci@0/pci@1/network@0    network-pciex8086,105e
IOBD/NET1    PCIE IOBD              /pci@780/pci@0/pci@1/network@0,1    network-pciex8086,105e
IOBD/PCIE    PCIE IOBD                   /pci@780/pci@0/pci@9/scsi@0         scsi-pciex1000,56 LSI,1064E
IOBD/NET2    PCIE IOBD                /pci@7c0/pci@0/pci@2/network@0    network-pciex8086,105e
IOBD/NET3    PCIE IOBD              /pci@7c0/pci@0/pci@2/network@0,1    network-pciex8086,105e
IOBD/PCIE0   PCIE    0            /pci@780/pci@0/pci@8/pci@0/TSI,mko                   TSI,mko    XVR200
IOBD/PCIX    PCIX IOBD              /pci@7c0/pci@0/pci@1/pci@0/isa@2                       isa
IOBD/PCIX    PCIX IOBD              /pci@7c0/pci@0/pci@1/pci@0/usb@5       usb-pciclass,0c0310
IOBD/PCIX    PCIX IOBD              /pci@7c0/pci@0/pci@1/pci@0/usb@6       usb-pciclass,0c0310
IOBD/PCIX    PCIX IOBD              /pci@7c0/pci@0/pci@1/pci@0/ide@8          ide-pci10b9,5229
IOBD/PCIE1   PCIE    1             /pci@7c0/pci@0/pci@8/pci@0/scsi@8           scsi-pci1000,30  LSI,1030
IOBD/PCIE1   PCIE    1           /pci@7c0/pci@0/pci@8/pci@0/scsi@8,1           scsi-pci1000,30  LSI,1030

# prtvtoc /dev/dsk/c0t2d0s1
prtvtoc: /dev/rdsk/c0t2d0s1: Unable to read Disk geometry errno = 0x5

#/dev/dsk/c0t2d0s0      /dev/rdsk/c0t2d0s0      /u01    ufs     2       yes     -
#/dev/dsk/c0t2d0s1      /dev/rdsk/c0t2d0s1      /u04    ufs     2       yes     -
/dev/md/dsk/d0  /dev/md/rdsk/d0 /       ufs     1       no      -
#/dev/md/dsk/d111       /dev/md/rdsk/d111       /u02    ufs     2       yes     -
#/dev/md/dsk/d121       /dev/md/rdsk/d121       /u03    ufs     2       yes     -

I certainly appreciate any help anyone can provide.

Thanks,
-Travis

---------- Post updated at 03:39 PM ---------- Previous update was at 02:19 PM ----------

Looks like there was an external POD connected to the server that was powered off after the outage and didn't get turned back on, which actually contained the /u01 mounts.

Now my question would be if not in the /etc/vfstab then where were these mount points that I overlooked?

# mount
/u01 on u1 read/write/setuid/devices/exec/xattr/atime/dev=4010002 on Tue Jan 18 10:13:57 2000
/u02 on u2 read/write/setuid/devices/exec/xattr/atime/dev=4010003 on Tue Jan 18 10:13:57 2000
/u03 on u3 read/write/setuid/devices/exec/xattr/atime/dev=4010004 on Tue Jan 18 10:13:57 2000
/u04 on u4 read/write/setuid/devices/exec/xattr/atime/dev=4010005 on Tue Jan 18 10:13:57 2000

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@0,0
       1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@1,0
       2. c0t2d0 <drive type unknown>
          /pci@780/pci@0/pci@9/scsi@0/sd@2,0
       3. c0t3d0 <drive type unknown>
          /pci@780/pci@0/pci@9/scsi@0/sd@3,0
       4. c2t10d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8/sd@a,0
       5. c2t11d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8/sd@b,0
       6. c3t8d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8,1/sd@8,0
       7. c3t9d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8,1/sd@9,0

hicksd8 · November 15, 2012, 2:27pm

Well this may need some investigation.

If the drive(s) are "offline" or "reserved" after a crash then they could just have been left in an inconsistent state.

A SCSI drive is a highly programmable device with a large number of registers held on what are technically known as "mode pages". A mode page can be set to "reserve" a drive to a particular host in a cluster, tell the drive to behave as if it has completely different geometry.

The first thing to do is to invoke the format command in expert mode:

format -e

and select a misbehaving drive and select "set all mode pages to default" from the menu. That will reset any flags left in an abnormal state. After that quit out.

Invoke format again in normal mode and see if you can select the drive and read the vtoc and see what slices are configured. If it won't let you do that then further investigation is required.

MobileGSP · November 15, 2012, 2:46pm

Same problem.

Although I think these disks probably died a long time ago and were replaced with external storage.

My new question based on this was, all of this new storage was mounted to "u1 u2 u3 u4" but it doesn't show the block mapping, and i'm not sure where it was set to automount if not in /etc/vfstab?

# format -e
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@0,0
       1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@1,0
       2. c0t2d0 <drive type unknown>
          /pci@780/pci@0/pci@9/scsi@0/sd@2,0
       3. c0t3d0 <drive type unknown>
          /pci@780/pci@0/pci@9/scsi@0/sd@3,0
       4. c2t10d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8/sd@a,0
       5. c2t11d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8/sd@b,0
       6. c3t8d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8,1/sd@8,0
       7. c3t9d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
          /pci@7c0/pci@0/pci@8/pci@0/scsi@8,1/sd@9,0
Specify disk (enter its number): 2



AVAILABLE DRIVE TYPES:
        0. Auto configure
        1. Quantum ProDrive 80S
        2. Quantum ProDrive 105S
        3. CDC Wren IV 94171-344
        4. SUN0104
        5. SUN0207
        6. SUN0327
        7. SUN0340
        8. SUN0424
        9. SUN0535
        10. SUN0669
        11. SUN1.0G
        12. SUN1.05
        13. SUN1.3G
        14. SUN2.1G
        15. SUN2.9G
        16. Zip 100
        17. Zip 250
        18. Peerless 10GB
        19. SUN72G
        20. SUN146G
        21. other
Specify disk type (enter its number):

hicksd8 · November 15, 2012, 3:04pm

Automounter config tables live in /etc and have names beginning:

auto<underscore>

followed by "master", "direct", "home" and stuff like that.

eg. auto_master

Look at these.

MobileGSP · November 15, 2012, 4:01pm

I found the auto_master and auto_home files, but neither have entries that I would expect to be mounting u1 u2 u3 u4

# ls /etc |grep auto
auto_home
auto_master

#cat /etc/auto_master
# Master map for automounter
#
+auto_master
/net            -hosts          -nosuid,nobrowse
/home           auto_home       -nobrowse

#cat /etc/auto_home
# Home directory map for automounter
#
+auto_home

I'm looking for these mounts

#mount
/u01 on u1 read/write/setuid/devices/exec/xattr/atime/dev=4010002 on Tue Jan 18 10:13:57 2000
/u02 on u2 read/write/setuid/devices/exec/xattr/atime/dev=4010003 on Tue Jan 18 10:13:57 2000
/u03 on u3 read/write/setuid/devices/exec/xattr/atime/dev=4010004 on Tue Jan 18 10:13:57 2000
/u04 on u4 read/write/setuid/devices/exec/xattr/atime/dev=4010005 on Tue Jan 18 10:13:57 2000

hicksd8 · November 15, 2012, 4:21pm

Now I've seen you last post I realise that these ARE mounted.

Google for "Solaris /u01" tells me these are Oracle database mounts.

unix - Why are Oracle directories named /u01 /u02 etc...? - Programmers

I'm no Oracle expert but I do know that some Oracle db implementations can use raw drives ie, not on filesystems.

It's possible that /u01, /u02, etc are nothing to do with your non-working drives but associated with the Oracle database product on that system.

Dunno......you need an Oracle expert next.

You could send a message to moderators on their specific forum on this site and ask where they recommend to post this to reach the expertise.

Good luck.

MobileGSP · November 15, 2012, 4:25pm

Yes these are Oracle Mounts, but I guess Solaris and Oracle interact differently than RedHat and Oracle.

In RedHat the directories that get mounted for /u01 ... are actually mount points in /etc/vfstab even when it's off external storage. Now the LUNs that get allocated to Oracle don't ever actually get mounted and would just be rawdevices.

I'm not longer down since they are now online, this was merely to try and figure out how Solaris worked with these mount points. It may be Oracle handling it in Solaris which is why I was so confused.

bartus11 · November 17, 2012, 4:57am

Those are ZFS filesystems, stored on separate ZFS pools. You can confirm it by running:

zpool list
zpool status
zfs list