Luxadm display not showing a LUN as expected

ludiegu · June 24, 2016, 5:25am

Hi all, I have the following LUN which is showing OK with its two paths:

DEVICE PROPERTIES for disk: /dev/rdsk/c23t500507630A3BC579d123s2
  Vendor:               IBM
  Product ID:           2107900
  Revision:             3010
  Serial Num:           75WB0114900
  Unformatted capacity: 30720.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0x16
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c23t500507630A3BC579d123s2
  /devices/pci@700/pci@2/pci@0/pci@3/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a3bc579,7b:c,raw
    LUN path port WWN:          500507630a3bc579
    Host controller port WWN:   21000024ff45d3d7
    Path status:                O.K.
  /dev/rdsk/c14t500507630A1BC579d123s2
  /devices/pci@600/pci@1/pci@0/pci@4/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a1bc579,7b:c,raw
    LUN path port WWN:          500507630a1bc579
    Host controller port WWN:   21000024ff45d2bd
    Path status:                O.K.

Now, I have instead this LUN, on the same server, which is a strange state:

root@MYHOST# luxadm display /dev/rdsk/c14t500507630A1BC579d124s2
DEVICE PROPERTIES for disk: /dev/rdsk/c14t500507630A1BC579d124s2
  Vendor:               IBM
  Product ID:           2107900
  Revision:             3010
  Serial Num:           75WB0116146
  Unformatted capacity: 512000.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0x16
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c14t500507630A1BC579d124s2
  /devices/pci@600/pci@1/pci@0/pci@4/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a1bc579,7c:c,raw
    LUN path port WWN:          500507630a1bc579
    Host controller port WWN:   21000024ff45d2bd
    Path status:                O.K.

root@MYHOST# luxadm display /dev/rdsk/c23t500507630A3BC579d124s2
DEVICE PROPERTIES for disk: /dev/rdsk/c23t500507630A3BC579d124s2
  Vendor:               IBM
  Product ID:           2107900
  Revision:             3010
  Serial Num:           75WB0116146
  Unformatted capacity: 512000.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0x16
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c23t500507630A3BC579d124s2
  /devices/pci@700/pci@2/pci@0/pci@3/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a3bc579,7c:c,raw
    LUN path port WWN:          500507630a3bc579
    Host controller port WWN:   21000024ff45d3d7
    Path status:                O.K.

Basically the paths are not aggregated as the LUN above, and I cannot understand why, to be honest. As you can see the Serial Number for that LUN is identical on both paths.
Any help is very welcome, thanks!
--
Diego

gull04 · June 24, 2016, 6:02am

Hi Diego,

What does ># uname -a return.

What does the out put of ># echo | format look like for the drive.

Regards

Gull04

ludiegu · June 24, 2016, 6:15am

Hi gull04,
here the output:

uname -a
SunOS es0bzsq5 5.10 Generic_150400-11 sun4v sparc sun4v

echo|format|grep c14t500507630A1BC579d124
     204. c14t500507630A1BC579d124 <IBM-2107900-3010 cyl 63998 alt 2 hd 64 sec 256>

echo|format|grep c23t500507630A3BC579d124
     502. c23t500507630A3BC579d124 <IBM-2107900-3010 cyl 63998 alt 2 hd 64 sec 256>

Thanks!

Peasant · June 24, 2016, 6:54am

Please post entire format output for one disk without grep (working and not working)

Multipath enabled devices will have under the number and device name /scsi_vhci/... while other will have hardware path /pci ... or similar.

Are you sure you presented the LUN on both paths ?
Have in installed additional FC cards in the meantime ?

Does :

mpathadm list lu

for the specified (no working) disk one path or there is no output for that disk ?

Hope that helps.
Best regards
Peasant.

achenle · June 24, 2016, 7:42am

This is also very useful:

-bash-3.00# for port in `fcinfo hba-port | grep Port | awk '{ print $4 }'`; do
> fcinfo remote-port -ls -p $port
> done

That will show you the targets and LUNs each HBA initiator port on your server can access.

ludiegu · June 24, 2016, 7:44am

Hi Peasant, here the output you asked:

working LUN
 501. c23t500507630A3BC579d123 <IBM-2107900-.166 cyl 3838 alt 2 hd 64 sec 256>
          /pci@700/pci@2/pci@0/pci@3/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a3bc579,7b
 203. c14t500507630A1BC579d123 <IBM-2107900-.166 cyl 3838 alt 2 hd 64 sec 256>
          /pci@600/pci@1/pci@0/pci@4/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a1bc579,7b

not working LUN
   204. c14t500507630A1BC579d124 <IBM-2107900-3010 cyl 63998 alt 2 hd 64 sec 256>
          /pci@600/pci@1/pci@0/pci@4/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a1bc579,7c
   502. c23t500507630A3BC579d124 <IBM-2107900-3010 cyl 63998 alt 2 hd 64 sec 256>
          /pci@700/pci@2/pci@0/pci@3/pci@0/pci@2/SUNW,qlc@0,1/fp@0,0/ssd@w500507630a3bc579,7c

The path are physical, as multipathing is done using Veritas software on those controllers.

Yes, as I can enquiry them via luxadm. No additional FC cards added in the meantime

There is not output for none of the disks I listed (both working and not working), as MPXIO is not enabled on those controllers.

Thanks!

gull04 · June 24, 2016, 8:22am

Hi Diego,

Just out of curiosity, how many LUNS are on the system all-together including internals if there are any?

Gull04

Peasant · June 24, 2016, 8:31am

Sorry, i have no idea about veritas multipathing solutions.
I only used native multipath on every linux/unix i worked on.

But, if you presented everything and MPXIO is not enabled..
You will see multiple devices (number depending on number of paths configured) for one presented LUN in the format output.

Also, check achenle suggestion.

*Edit achenle on cards which support NPIV the code you wrote will catch also Max NPIV Ports: resulting in output which is faulty (but will do no damage just print ILLEGAL WWN or alike)

Best regards
Peasant.

ludiegu · June 24, 2016, 8:36am

670ish LUNs + internal disks

gull04 · June 24, 2016, 10:25am

Hi Diego,

Your version of Solaris supports 16364 LUNS (I think), but do you know if there are any restrictions on the HBA's firmware.

It might be worth adding an other LUN if possible to see if the problem is related to that broken LUN only or if it affects any new LUNS as well to check if you have reached some kind of limit - or if the problem lies elsewhere.

Gull04

Peasant · June 24, 2016, 10:41am

Looks like operating system sees the lun on all paths normally (as device files are created), but luxadm command does not display correctly.
Could this be a bug of some sort ?

Does veritas multipath software recognize it as the same lun ?

Regards
Peasant.

achenle · June 24, 2016, 11:13am

What are the device names under /dev/vx for both the working and non-working LUNs?

ludiegu · June 24, 2016, 11:56am

@Gull04
I don't think so, as the same number of LUN is in an OK state on the other node of this cluster.

@Peasant
Correct, path are OK on format, but not on luxadm. As stated, the same OS is on the other node of the cluster, and the LUN is ok.
Veritas recognize the different paths of the "broken" disk as different ones, and here is my true problem

@achenle
The names are ibm_ds8x002_6146 for the "broken" one and ibm_ds8x002_4900 for the "correct" one.

Is there any way to "force" the luxadm infos?

Thanks!
--
Diego

ludiegu · June 29, 2016, 6:26am

For the record, I resolved doing a

luxadm -e offline

on one of the not-aggregated paths, and than did a simple

cfgadm -c configure

and

devfsadm

.