Removing Dead Paths And Unusable Path Solaris 9

jamba1 · August 1, 2011, 8:41pm

Hi,

One of our Solaris servers was peviously zoned and connected to 2 seperate arrays, one HP and the other EMC. The server is now only connected to the EMC. The sever has x2 single port HBA's.
When running cfgadm I see the following:

[

 
root@qwicprod /dev/rdsk]# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 fc-private connected configured unknown
c0::21000014c3503afe disk connected configured unknown
c0::500000e01097a811 disk connected configured unknown
c1 scsi-bus connected configured unknown
c1::dsk/c1t0d0 CD-ROM connected configured unknown
c2 fc-fabric connected configured unknown
c2::500601623b202f83 disk connected configured unknown <--- EMC
c2::5006016b3b202f83 disk connected configured unknown <--- EMC
c3 fc-fabric connected configured unknown
c3::50001fe15000de39 disk connected configured unusable <--- HP - don't know why it is picking this up
c3::50001fe15000de3b disk connected configured unusable <--- HP
c3::50001fe15000de3d disk connected configured unusable <--- HP
c3::50001fe15000de3f disk connected configured unusable <--- HP
c3::500601633b202f83 disk connected configured unknown <--- EMC
c3::5006016a3b202f83 disk connected configured unusable <--- EMC CX4BMC1_spa2_zone

I have checked the fibre switches and the server has been completely de-zoned from the HP array. We are also using powerpath which does not detect any dead paths. ie /etc/powermt config, /etc/powermt check and /etc/powermt dispaly dev=all. I have also noticed that when running format I see:

 
18. c3t50001FE15000DE3Bd110 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3b,6e
      19. c3t50001FE15000DE3Bd111 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3b,6f
      20. c3t50001FE15000DE3Bd120 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3b,78
      21. c3t50001FE15000DE3Bd121 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3b,79
      22. c3t50001FE15000DE3Bd130 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3b,82
      23. c3t50001FE15000DE3Bd131 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3b,83
      24. c3t50001FE15000DE3Dd110 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3d,6e
      25. c3t50001FE15000DE3Dd111 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3d,6f
      26. c3t50001FE15000DE3Dd120 <drive not available>
          /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w50001fe15000de3d,78
      27. c3t50001FE15000DE3Dd121 <drive not available>
truncated

Would using cfgadm to unconfigure be enough or is there something else that needs to run to clear up?

Also noticed that one of the paths connected to the EMC is showing unusable. Luxadm reports that all paths are good.

c3::5006016a3b202f83 disk connected configured unusable <--- EMC CX4BMC1_spa2_zone

 
[root@qwicprod /dev/rdsk]# luxadm -e port
Found path to 3 HBA ports
/devices/pci@8,600000/SUNW,qlc@1/fp@0,0:devctl CONNECTED
/devices/pci@8,600000/SUNW,qlc@2/fp@0,0:devctl CONNECTED
/devices/pci@9,600000/SUNW,qlc@2/fp@0,0:devctl CONNECTED
[root@qwicprod /dev/rdsk]# luxadm -e dump_map /devices/pci@8,600000/SUNW,qlc@2/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 b1a00 0 500601633b202f83 50060160bb202f83 0x0 (Disk device)
1 b1b00 0 5006016a3b202f83 50060160bb202f83 0x0 (Disk device)
2 c0900 0 210000e08b90318d 200000e08b90318d 0x1f (Unknown Type,Host Bus Adapter)
[root@qwicprod /dev/rdsk]# luxadm -e dump_map /devices/pci@8,600000/SUNW,qlc@1/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 11a00 0 500601623b202f83 50060160bb202f83 0x0 (Disk device)
1 11b00 0 5006016b3b202f83 50060160bb202f83 0x0 (Disk device)
2 21700 0 2100001b3281f2de 2000001b3281f2de 0x1f (Unknown Type,Host Bus Adapter)

[root@qwicprod /dev/rdsk]# luxadm display /dev/rdsk/c3t5006016A3B202F83d6s2 DEVICE PROPERTIES for disk: /dev/rdsk/c3t5006016A3B202F83d6s2
Vendor: DGC
Product ID: RAID 5
Revision: 0430
Serial Num: FCNCX0952006
Unformatted capacity: 51200.000 MBytes
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0x0
Device Type: Disk device
Path(s):
/dev/rdsk/c3t5006016A3B202F83d6s2
/devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w5006016a3b202f83,6:c,raw
LUN path port WWN: 5006016a3b202f83
Host controller port WWN: 210000e08b90318d
Path status: O.K. <-------------------------------------------- looks good
/dev/rdsk/c2t5006016B3B202F83d6s2
/devices/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w5006016b3b202f83,6:c,raw
LUN path port WWN: 5006016b3b202f83
Host controller port WWN: 2100001b3281f2de
Path status: O.K.
/dev/rdsk/c2t500601623B202F83d6s2
/devices/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601623b202f83,6:c,raw
LUN path port WWN: 500601623b202f83
Host controller port WWN: 2100001b3281f2de
Path status: O.K.
/dev/rdsk/c3t500601633B202F83d6s2
/devices/pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500601633b202f83,6:c,raw
LUN path port WWN: 500601633b202f83
Host controller port WWN: 210000e08b90318d
Path status: O.K.

[root@qwicprod /dev/rdsk]# /etc/powermt display Symmetrix logical device count=0 CLARiiON logical device count=8 Hitachi logical device count=0 Invista logical device count=0 HP xp logical device count=0 Ess logical device count=0 ==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
3073 pci@8,600000/SUNW,qlc@1/fp@0,0 optimal 16 0 - 2 0
3074 pci@8,600000/SUNW,qlc@2/fp@0,0 optimal 16 0 - 1 21 <---- errors

[root@qwicprod /dev/rdsk]# /etc/powermt display paths Symmetrix logical device count=0 ==============================================================================
----- Host Bus Adapters --------- ------ Storage System ----- - I/O Paths -
### HW Path ID Interface Total Dead
==============================================================================
CLARiiON logical device count=8
==============================================================================
----- Host Bus Adapters --------- ------ Storage System ----- - I/O Paths -
### HW Path ID Interface Total Dead
==============================================================================
3073 pci@8,600000/SUNW,qlc@1/fp@0,0 FCNCX095200695 SP A2 8 0
3073 pci@8,600000/SUNW,qlc@1/fp@0,0 FCNCX095200695 SP B3 8 0
3074 pci@8,600000/SUNW,qlc@2/fp@0,0 FCNCX095200695 SP A3 8 0
3074 pci@8,600000/SUNW,qlc@2/fp@0,0 FCNCX095200695 SP B2 8 0
[root@qwicprod /dev/rdsk]# /etc/powermt display port_mode
Storage class = Symmetrix
==============================================================================
----------- Storage System --------------- -- I/O Paths -- --- Stats ---
ID Interface Wt_Q Total Dead Q-IOs Mode
==============================================================================
Storage class = CLARiiON
==============================================================================
----------- Storage System --------------- -- I/O Paths -- --- Stats ---
ID Interface Wt_Q Total Dead Q-IOs Mode
==============================================================================
FCNCX095200695 SP A2 256 8 0 0 Enabled
FCNCX095200695 SP B3 256 8 0 0 Enabled
FCNCX095200695 SP A3 256 8 0 0 Enabled
FCNCX095200695 SP B2 256 8 0 0 Enabled

Could this path be reporting unusable due to the other dead paths still configured on the HP array.

Yeaboem · August 1, 2011, 8:47pm

have you tried 'devfsadm -C' to rebuild/clean-up your /dev tree?

jamba1 · August 1, 2011, 9:30pm

Hi Yeaboem

Yes I have tried to clean up using devfsadm -C but it makes no difference

Yeaboem · August 1, 2011, 10:47pm

ah, yes. you will have to cfgadm -c unconfigure ... the stale occupants first, then devfsadm -C to clean /dev. After you've done that, the format command will show only your active disks/luns.

jamba1 · August 2, 2011, 6:32pm

Ok Yeaboem thanks for confirming this. Is this safe to do on critical production system cfgadm -c unconfigure c3::50001fe15000de39

Yeaboem · August 2, 2011, 7:30pm

the answer is... Maybe?

Have you configured multipathing using both HBAs and pathways to your luns? I am concerned that issuing the cfgadm -c unconfigure command may result in brief intervals where activity is suspended on that HBA. I know for certain that 'cfgadm -c insert/remove' do interrupt traffic thru the HBA, I just don't remember perfectly if the 'configure/unconfigure' commands do, also. If you have multipathing configured, this is not a problem, as your luns would be accessed by an alternate path. My sense of it is that the configure/unconfigure should be safe on a running system.

jamba1 · August 2, 2011, 8:59pm

Yes we have x2 single port HBA's with multipathing configured. I will do the unconfigure during a quiet period. Thanks

 
[root@qwicprod /dev/rdsk]# luxadm -e dump_map /devices/pci@8,600000/SUNW,qlc@2/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 b1a00 0 500601633b202f83 50060160bb202f83 0x0 (Disk device)
1 b1b00 0 5006016a3b202f83 50060160bb202f83 0x0 (Disk device)
2 c0900 0 210000e08b90318d 200000e08b90318d 0x1f (Unknown Type,Host Bus Adapter)
[root@qwicprod /dev/rdsk]# luxadm -e dump_map /devices/pci@8,600000/SUNW,qlc@1/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 11a00 0 500601623b202f83 50060160bb202f83 0x0 (Disk device)
1 11b00 0 5006016b3b202f83 50060160bb202f83 0x0 (Disk device)
2 21700 0 2100001b3281f2de 2000001b3281f2de 0x1f (Unknown Type,Host Bus Adapter)