Problem with accessing SAN disks

michael.chow · May 21, 2008, 2:24am

Hi, I'm having a problem when attempting to define the OCR location for my 10g RAC setup on Solaris 10. I get the following error:

The specified shared raw partition /dev/did/rdsk/d1s0 may not have the correct permission. Verify that the partition is owned by Oracle user.

As per the Oracle10g installation guide, I did the following:

# chown root:dba /devices/pseudo/did@1:1,1s0,raw
# chmod 640 /devices/pseudo/did@1:1,1s0,raw
# ls -l /devices/pseudo/did@1:1,1s0,raw
crw-r----- 1 root dba 239, 32 May 20 12:37 /devices/pseudo/did@1:1,1s0,raw

After I have the above, the installer still complains of the same error.

Cheers.

michael.chow · May 21, 2008, 4:18am

I found the following entries in the system console log:

May 21 15:34:57 wrkarea1 scsi: [ID 107833 kern.warning] WARNING: /pci@1c,600000/SUNW,qlc@1
/fp@0,0/ssd@w256000c0ffd98fe5,7 (ssd4):
May 21 15:34:57 wrkarea1 Corrupt label; wrong magic number
May 21 15:34:57 wrkarea1 scsi: [ID 107833 kern.warning] WARNING: /pci@1c,600000/SUNW,qlc@1
/fp@0,0/ssd@w256000c0ffd98fe5,0 (ssd7):
May 21 15:34:57 wrkarea1 Corrupt label; wrong magic number
May 21 15:34:57 wrkarea1 scsi: [ID 107833 kern.warning] WARNING: /pci@1c,600000/SUNW,qlc@1
/fp@0,0/ssd@w256000c0ffd98fe5,0 (ssd7):
May 21 15:34:57 wrkarea1 Corrupt label; wrong magic number
May 21 15:34:58 wrkarea1 scsi: [ID 107833 kern.warning] WARNING: /pci@1d,700000/SUNW,qlc@2
/fp@0,0/ssd@w266000c0ffe98fe5,7 (ssd22):
May 21 15:34:58 wrkarea1 Corrupt label; wrong magic number
May 21 15:34:58 wrkarea1 scsi: [ID 107833 kern.warning] WARNING: /pci@1d,700000/SUNW,qlc@2
/fp@0,0/ssd@w266000c0ffe98fe5,7 (ssd22):
May 21 15:34:58 wrkarea1 Corrupt label; wrong magic number
May 21 15:34:58 wrkarea1 scsi: [ID 107833 kern.warning] WARNING: /pci@1d,700000/SUNW,qlc@2
/fp@0,0/ssd@w266000c0ffe98fe5,0 (ssd29):
May 21 15:34:58 wrkarea1 Corrupt label; wrong magic number

I have tried to run format to label the disks above but I can't find any of the above disks:

# format
Searching for disks...done

c0t256000C0FFD98FE5d0: configured with capacity of 10.25GB
c0t256000C0FFD98FE5d7: configured with capacity of 5.61GB

AVAILABLE DISK SELECTIONS:
0. c0t216000C0FF898FE5d0 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,0
1. c0t216000C0FF898FE5d1 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,1
2. c0t216000C0FF898FE5d2 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,2
3. c0t216000C0FF898FE5d3 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,3
4. c0t216000C0FF898FE5d4 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,4
5. c0t216000C0FF898FE5d5 <SUN-StorEdge3510-413C cyl 1498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,5
6. c0t216000C0FF898FE5d6 <SUN-StorEdge3510-413C cyl 9998 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,6
7. c0t216000C0FF898FE5d7 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,7
8. c0t216000C0FF898FE5d8 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,8
9. c0t216000C0FF898FE5d9 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,9
10. c0t216000C0FF898FE5d10 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,a
11. c0t216000C0FF898FE5d11 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,b
12. c0t216000C0FF898FE5d12 <SUN-StorEdge3510-413C cyl 1498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,c
13. c0t216000C0FF898FE5d13 <SUN-StorEdge3510-413C cyl 9998 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898fe5,d
14. c0t256000C0FFD98FE5d0 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,0
15. c0t256000C0FFD98FE5d1 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,1
16. c0t256000C0FFD98FE5d2 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,2
17. c0t256000C0FFD98FE5d3 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,3
18. c0t256000C0FFD98FE5d4 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,4
19. c0t256000C0FFD98FE5d5 <SUN-StorEdge3510-413C cyl 1498 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,5
20. c0t256000C0FFD98FE5d6 <SUN-StorEdge3510-413C cyl 9998 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,6
21. c0t256000C0FFD98FE5d7 <SUN-StorEdge3510-413C cyl 5749 alt 2 hd 64 sec 32>
/pci@1c,600000/SUNW,qlc@1/fp@0,0/ssd@w256000c0ffd98fe5,7
22. c1t226000C0FFB98FE5d0 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,0
23. c1t226000C0FFB98FE5d1 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,1
24. c1t226000C0FFB98FE5d2 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,2
25. c1t226000C0FFB98FE5d3 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,3
26. c1t226000C0FFB98FE5d4 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,4
27. c1t226000C0FFB98FE5d5 <SUN-StorEdge3510-413C cyl 1498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,5
28. c1t226000C0FFB98FE5d6 <SUN-StorEdge3510-413C cyl 9998 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,6
29. c1t226000C0FFB98FE5d7 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,7
30. c1t226000C0FFB98FE5d8 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,8
31. c1t226000C0FFB98FE5d9 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,9
32. c1t226000C0FFB98FE5d10 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,a
33. c1t226000C0FFB98FE5d11 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,b
34. c1t226000C0FFB98FE5d12 <SUN-StorEdge3510-413C cyl 1498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,c
35. c1t226000C0FFB98FE5d13 <SUN-StorEdge3510-413C cyl 9998 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w226000c0ffb98fe5,d
36. c1t266000C0FFE98FE5d0 <drive type unknown>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,0
37. c1t266000C0FFE98FE5d1 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,1
38. c1t266000C0FFE98FE5d2 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,2
39. c1t266000C0FFE98FE5d3 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,3
40. c1t266000C0FFE98FE5d4 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,4
41. c1t266000C0FFE98FE5d5 <SUN-StorEdge3510-413C cyl 1498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,5
42. c1t266000C0FFE98FE5d6 <SUN-StorEdge3510-413C cyl 9998 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,6
43. c1t266000C0FFE98FE5d7 <drive type unknown>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,7
44. c3t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1f,700000/scsi@2/sd@0,0
45. c3t2d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1f,700000/scsi@2/sd@2,0
46. c3t3d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1f,700000/scsi@2/sd@3,0
Specify disk (enter its number):

michael.chow · May 21, 2008, 9:21pm

I ran luxadm -e port and noticed that the disks that are having the error shows not connected:

# luxadm -e port
/devices/pci@1c,600000/SUNW,qlc@1/fp@0,0:devctl CONNECTED
/devices/pci@1c,600000/SUNW,qlc@1,1/fp@0,0:devctl NOT CONNECTED
/devices/pci@1d,700000/SUNW,qlc@2,1/fp@0,0:devctl NOT CONNECTED
/devices/pci@1d,700000/SUNW,qlc@2/fp@0,0:devctl CONNECTED

How do I connect these disks?

system · May 21, 2008, 9:43pm

The drives do show up in your format output:

36. c1t266000C0FFE98FE5d0 <drive type unknown>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,0
37. c1t266000C0FFE98FE5d1 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,1
38. c1t266000C0FFE98FE5d2 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,2
39. c1t266000C0FFE98FE5d3 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,3
40. c1t266000C0FFE98FE5d4 <SUN-StorEdge3510-413C cyl 10498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,4
41. c1t266000C0FFE98FE5d5 <SUN-StorEdge3510-413C cyl 1498 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,5
42. c1t266000C0FFE98FE5d6 <SUN-StorEdge3510-413C cyl 9998 alt 2 hd 64 sec 32>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,6
43. c1t266000C0FFE98FE5d7 <drive type unknown>
/pci@1d,700000/SUNW,qlc@2/fp@0,0/ssd@w266000c0ffe98fe5,7

system · May 21, 2008, 10:17pm

Check your fibre connection with:

/usr/sbin/cfgadm -al

If a controller is not seen, or seen as unconfigured run:

/usr/sbin/cfgadm -c configure c?

michael.chow · May 21, 2008, 10:24pm

I have a StorEdge 3510 and am running Solaris 10. When I tried to dd a disk to use as my Oracle OCR and Voting disk I get an error saying corrupt label - wrong magic number. So I did some extra checks and found the following:

# luxadm -e port
/devices/pci@1d,700000/SUNW,qlc@2/fp@0,0:devctl                    CONNECTED
/devices/pci@1c,600000/SUNW,qlc@1/fp@0,0:devctl                    CONNECTED
/devices/pci@1c,600000/SUNW,qlc@1,1/fp@0,0:devctl                  NOT CONNECTED
/devices/pci@1d,700000/SUNW,qlc@2,1/fp@0,0:devctl                  NOT CONNECTED
# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             fc-fabric    connected    configured   unknown
c0::210000e08b8683cc           unknown      connected    unconfigured unknown
c0::216000c0ff898fe5           disk         connected    configured   unknown
c0::256000c0ffd98fe5           disk         connected    configured   unknown
c1                             fc-fabric    connected    configured   unknown
c1::210000e08b8656ca           unknown      connected    unconfigured unknown
c1::226000c0ffb98fe5           disk         connected    configured   unknown
c1::266000c0ffe98fe5           disk         connected    configured   unknown
c2                             scsi-bus     connected    configured   unknown
c2::dsk/c2t0d0                 CD-ROM       connected    configured   unknown
c3                             scsi-bus     connected    configured   unknown
c3::dsk/c3t0d0                 disk         connected    configured   unknown
c3::dsk/c3t2d0                 disk         connected    configured   unknown
c3::dsk/c3t3d0                 disk         connected    configured   unknown
c4                             fc           connected    unconfigured unknown
c5                             fc           connected    unconfigured unknown
c6                             scsi-bus     connected    unconfigured unknown
usb0/1                         usb-kbd      connected    configured   ok
usb0/2                         usb-mouse    connected    configured   ok
usb1/1                         unknown      empty        unconfigured ok
usb1/2                         unknown      empty        unconfigured ok

I then did these steps:

# cfgadm -c configure c4
# cfgadm -c configure c5
# cfgadm -c configure c6
# reboot -- -r

But after the server rebooted, running luxadm -e port and cfgadm -al still gave me the same results. How to I make the disks show as configured in cfgadm and connected in luxadm -e port? I have already checked and verified that the physical connections are in place.

system · May 21, 2008, 10:36pm

Starting multiple threads about the same problem isn't going to speed up your resolution.

Have a look at Info Doc 89031 on sunsolve to help you trouble shoot your problem.

michael.chow · May 21, 2008, 10:50pm

Sorry about the multiple thread. Will look at the document in sunsolve. Cheers.

michael.chow · May 21, 2008, 11:02pm

I get this error when trying to format the disks:

selecting c1t266000C0FFE98FE5d0
[disk formatted]
Error occurred with device in use checking: No such device

selecting c1t266000C0FFE98FE5d7
[disk formatted]
Error occurred with device in use checking: No such device

linuxaddict7 · May 21, 2008, 11:14pm

run devfsadm and see if it recreates those devices again.

michael.chow · May 21, 2008, 11:44pm

I've solved it. Had to format the disks separately from the individual nodes. Thanks a lot.

DukeNuke2 · May 22, 2008, 3:32am

what server hardware do you use? and what is the output of format? also the output of "prtdiag -v" is needed...

michael.chow · May 22, 2008, 3:46am

I've got this problem solved. Thanks a lot.

system · May 22, 2008, 8:40am

How did you fix this problem ?

# luxadm -e port
/devices/pci@1c,600000/SUNW,qlc@1/fp@0,0:devctl CONNECTED
/devices/pci@1c,600000/SUNW,qlc@1,1/fp@0,0:devctl NOT CONNECTED
/devices/pci@1d,700000/SUNW,qlc@2,1/fp@0,0:devctl NOT CONNECTED
/devices/pci@1d,700000/SUNW,qlc@2/fp@0,0:devctl CONNECTED

format won't fix that.

system · May 22, 2008, 8:41am

FYI... Here is his other thread

reborg · May 22, 2008, 8:50am

Thanks Tornado. Threads merged.