MPxIO Multipathing Problem

UNIX Solaris
1

Hi,

I believe this is an OS multipathing issue - Solaris 10

We have x2 Dual port 8GB Qlogic HBA's installed in our SOlaris 10 host.

HBA0 (IOU0) Port WWN's - 09f4 (pci@3,700000/SUNW,qlc@0)
09f5 (pci@3,700000/SUNW,qlc@0,1)

HBA1 (IOU1) Port WWN's - 09fc (pci@13,700000/SUNW,qlc@0)
09fd (pci@13,700000/SUNW,qlc@0,1)

HBA0 goes to FABRIC A (Fibre channel switch ports 60 and 12)
HBA1 goes to FABRIC B (Fibre channel switch ports 60 and 12)

The zoning on both fabrics has been checked and is correct. The zoning configuration allows:

HBA0 to see EMC array ports a0 - 50:06:01:60:3b:20:f3:83
a2 - 62:3b
b1 - 69:3b
b3 - 6b:3b

HBA1 to see EMC array ports a1 - 50:06:01:61:3b:20:f3:83
a3 - 63:3b
b0 - 68:3b
b2 - 6a:3b

MPxIO was enabled using stmsboot -e when the server was originally built. ALUA configured on the EMC SAN.

The problem that I have is that only x4 paths from HBA0 (IOU0) are detected and have paths enabled to the SAN disks See output below, I would expect to also see the other paths from HBA1(IOU1) for initiator port name and its target ports (61:3b, 63:3b, 68:3b, 6a:3b) The target port groups for failover are mapped from the same HBA

Where is the other HBA and why is is not detected through the mpath
daemon ?

 
# mpathadm show lu /dev/rdsk/c6t600601606519250060CCBED4F6B1E011d0s2
Logical Unit:  /dev/rdsk/c6t600601606519250060CCBED4F6B1E011d0s2
        mpath-support:  libmpscsi_vhci.so
        Vendor:  DGC
        Product:  RAID 5
        Revision:  0430
        Name Type:  unknown type
        Name:  600601606519250060ccbed4f6b1e011
        Asymmetric:  yes
        Current Load Balance:  round-robin
        Logical Unit Group ID:  NA
        Auto Failback:  on
        Auto Probing:  NA
        Paths:
                Initiator Port Name:  21000024ff3209f4
                Target Port Name:  500601603b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff3209f4
                Target Port Name:  500601623b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff3209f4
                Target Port Name:  5006016b3b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff3209f4
                Target Port Name:  500601693b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
        Target Port Groups:
                ID:  1
                Explicit Failover:  yes
                Access State:  active not optimized
                Target Ports:
                        Name:  500601603b202f83
                        Relative ID:  1
                        Name:  500601623b202f83
                        Relative ID:  3
                ID:  2
                Explicit Failover:  yes
                Access State:  active optimized
                Target Ports:
                        Name:  5006016b3b202f83
                        Relative ID:  10
                        Name:  500601693b202f83
                        Relative ID:  8

Some additional information.

 
# mpathadm list lu | grep -i "Operational Path Count:"
Operational Path Count: 4 <--- for every SAN disk
 
HBA0
----
# fcinfo remote-port -sl -p 21000024ff3209f4 | more  
Remote Port WWN: 500601603b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 3
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250060CCBED4F6B1E011d0s2
        LUN: 1
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250061CCBED4F6B1E011d0s2
        LUN: 2
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250062CCBED4F6B1E011d0s2
        LUN: 3
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925002464C1DF3446E111d0s2
        LUN: 4
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250036A2EE174346E111d0s2
        LUN: 5
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250088A10035B84BE111d0s2
        LUN: 6
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500AA8A0F382195DF11d0s2
        LUN: 7
          Vendor: DGC
          Product: RAID 5
        OS Device Name: /dev/rdsk/c6t6006016065192500842F1A702395DF11d0s2
        LUN: 8
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500724AB0ED5C79E111d0s2
        LUN: 9
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925000CB72E0E5D79E111d0s2
        LUN: 10
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250018D58D2C626FDF11d0s2
        LUN: 11
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500C8C883375FA0DF11d0s2
        LUN: 12
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925005A1ADB9D5EA0DF11d0s2
Remote Port WWN: 500601623b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 3
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250060CCBED4F6B1E011d0s2
        LUN: 1
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250061CCBED4F6B1E011d0s2
        LUN: 2
          Vendor: DGC
        Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250062CCBED4F6B1E011d0s2
        LUN: 3
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925002464C1DF3446E111d0s2
        LUN: 4
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250036A2EE174346E111d0s2
        LUN: 5
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250088A10035B84BE111d0s2
        LUN: 6
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500AA8A0F382195DF11d0s2
        LUN: 7
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500842F1A702395DF11d0s2
        LUN: 8
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500724AB0ED5C79E111d0s2
        LUN: 9
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925000CB72E0E5D79E111d0s2
        LUN: 10
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250018D58D2C626FDF11d0s2
        LUN: 11
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500C8C883375FA0DF11d0s2
        LUN: 12
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925005A1ADB9D5EA0DF11d0s2
Remote Port WWN: 500601693b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250060CCBED4F6B1E011d0s2
        LUN: 1
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250061CCBED4F6B1E011d0s2
        LUN: 2
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250062CCBED4F6B1E011d0s2
        LUN: 3
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925002464C1DF3446E111d0s2
        LUN: 4
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250036A2EE174346E111d0s2
        LUN: 5
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250088A10035B84BE111d0s2
        LUN: 6
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500AA8A0F382195DF11d0s2
        LUN: 7
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500842F1A702395DF11d0s2
        LUN: 8
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500724AB0ED5C79E111d0s2
        LUN: 9
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925000CB72E0E5D79E111d0s2
        LUN: 10
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250018D58D2C626FDF11d0s2
        LUN: 11
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500C8C883375FA0DF11d0s2
        LUN: 12
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925005A1ADB9D5EA0DF11d0s2
Remote Port WWN: 5006016b3b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 6
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250060CCBED4F6B1E011d0s2
        LUN: 1
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250061CCBED4F6B1E011d0s2
        LUN: 2
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250062CCBED4F6B1E011d0s2
        LUN: 3
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925002464C1DF3446E111d0s2
        LUN: 4
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250036A2EE174346E111d0s2
        LUN: 5
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250088A10035B84BE111d0s2
        LUN: 6
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500AA8A0F382195DF11d0s2
        LUN: 7
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500842F1A702395DF11d0s2
        LUN: 8
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500724AB0ED5C79E111d0s2
        LUN: 9
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925000CB72E0E5D79E111d0s2
        LUN: 10
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t600601606519250018D58D2C626FDF11d0s2
        LUN: 11
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t6006016065192500C8C883375FA0DF11d0s2
        LUN: 12
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c6t60060160651925005A1ADB9D5EA0DF11d0s2

HBA1 - unable to detect SAN disks but can see SAN ports 61:3b, 63:3b, 68:3b, 6a:3b

root@oradb1 # fcinfo remote-port -sl -p 21000024ff3209fc
Remote Port WWN: 500601613b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 3
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: LUNZ
          OS Device Name: /dev/rdsk/c3t500601613B202F83d0s2
Remote Port WWN: 500601633b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 1
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: LUNZ
          OS Device Name: /dev/rdsk/c3t500601633B202F83d0s2
Remote Port WWN: 500601683b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 1
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: LUNZ
          OS Device Name: /dev/rdsk/c3t500601683B202F83d0s2
Remote Port WWN: 5006016a3b202f83
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160bb202f83
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: LUNZ
          OS Device Name: /dev/rdsk/c3t5006016A3B202F83d0s2

Just for comparison below output is from another Solaris server only this is running Solaris 11. Zoning configuration is the same as above albeit the HBA's are single port. Both HBA's and all paths to SAN detected.

 
# mpathadm list lu | grep -i "Operational Path Count:"
Operational Path Count: 8
 
 
# mpathadm show lu /dev/rdsk/c0t60060160651925008AAA53E34C22E211d0s2
Logical Unit:  /dev/rdsk/c0t60060160651925008AAA53E34C22E211d0s2
        mpath-support:  libmpscsi_vhci.so
        Vendor:  DGC
        Product:  VRAID
        Revision:  0430
        Name Type:  unknown type
        Name:  60060160651925008aaa53e34c22e211
        Asymmetric:  yes
        Current Load Balance:  round-robin
        Logical Unit Group ID:  NA
        Auto Failback:  on
        Auto Probing:  NA
        Paths:
                Initiator Port Name:  21000024ff27c1d6
                Target Port Name:  5006016b3b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff27c1d6
                Target Port Name:  500601693b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff277a62
                Target Port Name:  500601683b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff277a62
                Target Port Name:  5006016a3b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff27c1d6
                Target Port Name:  500601603b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff27c1d6
                Target Port Name:  500601623b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff277a62
                Target Port Name:  500601613b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
                Initiator Port Name:  21000024ff277a62
                Target Port Name:  500601633b202f83
                Override Path:  NA
                Path State:  OK
                Disabled:  no
        Target Port Groups:
                ID:  2
                Explicit Failover:  yes
                Access State:  active not optimized
                Target Ports:
                        Name:  5006016b3b202f83
                        Relative ID:  10
                        Name:  500601693b202f83
                        Relative ID:  8
                        Name:  500601683b202f83
                        Relative ID:  7
                        Name:  5006016a3b202f83
                        Relative ID:  9
                ID:  1
                Explicit Failover:  yes
                Access State:  active optimized
                Target Ports:
                        Name:  500601603b202f83
                        Relative ID:  1
                        Name:  500601623b202f83
                        Relative ID:  3
                        Name:  500601613b202f83
                        Relative ID:  2
                        Name:  500601633b202f83
                        Relative ID:  4

Has anyone come across a similar issue or advise.

Thanks in advance

2

I recall a recent chain about HBAs and MPIO that finally went into the direction of leveling the network connections they were on, as MPIO secifically did not do HBAs, at least for their version. All good questions come around regularly. Did you google the site?

3

Turns out that the second HBA for some unknown reason had lost connectivity to the SAN and was not reporting on the usual connectivity status on the array. HBA had to be re-registered to pick up alternate paths.