SAN Disk Appearing double in AIX

Hello Folks,

I have directly connected my IBM Pseries AIX machine to SAN Storage

I have a dual port Fibre Channel Adapter
Connected both fiber ports to SAN Box IBM DS4500 Controller A & Controller B

Using IBM DS storage manager client 10 -- created one logical disk and assigned to a default group

on my AIX machine

# oslevel -s
6100-05-00-0000

# lsdev -Cc disk
hdisk8 Available 0A-08-02     MPIO Other DS4K Array Disk
hdisk9 Available 0A-08-02     MPIO Other DS4K Array Disk

# lsdev -Cc adapter

fcs0      Available 0A-08 FC Adapter
fcs1      Available 0A-09 FC Adapter

I see two disks in my AIX machine HDISK8 and HDISK9 is same disk but appearing twice . I have disconnected one cable from the controller , deleted hdisk9 and ran cfgmgr , I don't see hdisk9

Do I need to have SAN Switch inorder for it to appear only once or is there any other configuration which I am missing.... ?

THanks.

Hmmm,
Looking at the output it seems both the disk are seen by fcs0, can you paste the output of

lspath -H

Check the setting on interface of your fibre adapter.

lsattr -El fscsiX, where x=0 and 1

Is dynamic tracking set to yes?

Also, I see your AIX OS version is 6.1TL05, you are at basic level of TL05, problem like yours may have been resolved in higher TL or SP levels, I recommend upgrade your OS as well.

Can you paste the output of

lspv

If none appears in 2nd field, then run

chdev -l hdisk8 -a pv=yes, and same to hdisk9

Do you see the matching PVID no?

:slight_smile: I am jealous - your own SAN!

:slight_smile: no, i got this test san box for testing oracle applications before deploying rman backup.

ok.

Port 0 = FCS0 of Dual Port Fibre Card connected to Controller A
Port 1 = FCS1 of Dual Port Fibre Card connected to Controller B

DS Storage Client Software, in mappings, I can see only one Default Group to which I assigned one disk.

# lspath -H
status  name   parent

Enabled hdisk8 fscsi0
Enabled dac0   fscsi0
Enabled hdisk9 fscsi1
Enabled dac1   fscsi1

# lsattr -El fscsi0
attach       al           How this adapter is CONNECTED         False
dyntrk       no           Dynamic Tracking of FC Devices        True
fc_err_recov delayed_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x1          Adapter SCSI ID                       False
sw_fc_class  3            FC Class for Fabric                   True

# lsattr -El fscsi1
attach       al           How this adapter is CONNECTED         False
dyntrk       no           Dynamic Tracking of FC Devices        True
fc_err_recov delayed_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x1          Adapter SCSI ID                       False
sw_fc_class  3            FC Class for Fabric                   True

# lspv

hdisk8          none                                None
hdisk9          none                                None


# chdev -l hdisk8 -a pv=yes
hdisk8 changed
# chdev -l hdisk9 -a pv=yes
hdisk8 changed

# lspv

hdisk8          00c7780e803680fd                    None
hdisk9          00c7780e803680fd                    None

# lscfg -vl fcs0
  fcs0             U787B.001.DNW3313-P1-C1-T1  FC Adapter

        Part Number.................03N5029
        EC Level....................A
        Serial Number...............1F7050C060
        Manufacturer................001F
        Customer Card ID Number.....5759
        FRU Number.................. 03N5029
        Device Specific.(ZM)........3
        Network Address.............10000000C9612EFA
        ROS Level and ID............02C82774
        Device Specific.(Z0)........1036406D
        Device Specific.(Z1)........00000000
        Device Specific.(Z2)........00000000
        Device Specific.(Z3)........03000909
        Device Specific.(Z4)........FFC01231
        Device Specific.(Z5)........02C82774
        Device Specific.(Z6)........06C12715
        Device Specific.(Z7)........07C12774
        Device Specific.(Z8)........20000000C9612EFA
        Device Specific.(Z9)........BS2.71X4
        Device Specific.(ZA)........B1F2.70A5
        Device Specific.(ZB)........B2F2.71X4
        Device Specific.(ZC)........00000000
        Hardware Location Code......U787B.001.DNW3313-P1-C1-T1


# lscfg -vl fcs1
  fcs1             U787B.001.DNW3313-P1-C1-T2  FC Adapter

        Part Number.................03N5029
        EC Level....................A
        Serial Number...............1F7050C060
        Manufacturer................001F
        Customer Card ID Number.....5759
        FRU Number.................. 03N5029
        Device Specific.(ZM)........3
        Network Address.............10000000C9612EF9
        ROS Level and ID............02C82774
        Device Specific.(Z0)........1036406D
        Device Specific.(Z1)........00000000
        Device Specific.(Z2)........00000000
        Device Specific.(Z3)........03000909
        Device Specific.(Z4)........FFC01231
        Device Specific.(Z5)........02C82774
        Device Specific.(Z6)........06C12715
        Device Specific.(Z7)........07C12774
        Device Specific.(Z8)........20000000C9612EF9
        Device Specific.(Z9)........BS2.71X4
        Device Specific.(ZA)........B1F2.70A5
        Device Specific.(ZB)........B2F2.71X4
        Device Specific.(ZC)........00000000
        Hardware Location Code......U787B.001.DNW3313-P1-C1-T2


What version of AIX are you running? Have you installed the appropriate software to merge the disks together?

We have a 5.1 system that blends together two or more hdisks with the same serial number as a vpath. We use the command lsvpcfg to display what is matched up. We then add the vpath device to the volume group.

Just a snippet:-

RBATTE1 @ /usr/home/RBATTE1>lsvpcfg
vpath1 (Avail pv db1vg02) 50715182 = hdisk688 (Avail ) hdisk906 (Avail )
vpath2 (Avail pv db1vg02) 50815182 = hdisk689 (Avail ) hdisk907 (Avail )
vpath3 (Avail pv db1vg02) 50915182 = hdisk690 (Avail ) hdisk908 (Avail )
etc.
etc.

RBATTE1 @ /usr/home/RBATTE1>lsvg -p db1vg02 
db1vg02:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
vpath1            active            521         3           00..00..00..00..03
vpath2            active            521         1           00..00..00..00..01
vpath3            active            521         1           00..00..00..00..01
vpath4            active            521         1           00..00..00..00..01
vpath5            active            521         1           00..00..00..00..01
vpath6            active            521         1           00..00..00..00..01
vpath7            active            521         1           00..00..00..00..01

If you are newer, then you would need MPIO, which I'm not familiar with, but I think that would blend multiple-path devices into hdisks, so it would seem more normal.

I would also recommend splitting your fibre connections. If you lose a path and feel that you have to replace the fibre card, then you are looking at downtime (well all SAN disks off-line anyway) to do so. If it really is for testing, then splitting would still be a good idea as you could pull a cable and see how it all reacts. :eek:

Robin
Liverpool/Blackburn
UK

Ok,
Have you install the ODM file set for type of storage? It's a native software of aix to detect type of disk.

I am not sure, if you company has bought MPIO software, its a licensed product (its free with v7000 storage), with DS4000 series, you better run by your manager. If yes you will install the devices.sddpcm file set for MPIO.

Set the dynamic tracking to yes and fc_err_recov to fast_fail, if your rootvg is local disk or vscsi, you can run the command without -P flag. else use -P flag and reboot the box for the changes to be affective.

I am using VIOS Version

$ ioslevel
2.1.3.10-FP23

$ oem_setup_env
#
# oslevel -s
6100-05-00-0000

so, in VIOS as far as I know there is no SDD

I do have the MPIO drivers installed which came with the VIOS

# lsdev -Cc disk
hdisk0 Available 09-08-00-3,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 09-08-00-4,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 09-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk3 Available 09-08-00-8,0 16 Bit LVD SCSI Disk Drive
hdisk4 Available 09-08-01-3,0 16 Bit LVD SCSI Disk Drive
hdisk5 Available 09-08-01-4,0 16 Bit LVD SCSI Disk Drive
hdisk6 Available 09-08-01-5,0 16 Bit LVD SCSI Disk Drive
hdisk7 Available 09-08-01-8,0 16 Bit LVD SCSI Disk Drive
hdisk8 Available 0A-08-02     MPIO Other DS4K Array Disk
hdisk9 Available 0A-09-02     MPIO Other DS4K Array Disk
#
# lsvpcfg
ksh: lsvpcfg:  not found.



# lslpp -l | grep mpio
  devices.common.IBM.mpio.rte
  devices.common.IBM.mpio.rte

# lslpp -lc | grep mpio
/usr/lib/objrepos:devices.common.IBM.mpio.rte:6.1.5.0::COMMITTED:I:MPIO Disk Path Control Module :
/etc/objrepos:devices.common.IBM.mpio.rte:6.1.5.0::COMMITTED:I:MPIO Disk Path Control Module :


# manage_disk_drivers -l
Device           Present Driver     Driver Options
2810XIV          AIX_AAPCM          AIX_AAPCM,AIX_non_MPIO
DS4100           AIX_APPCM          AIX_APPCM,AIX_fcparray
DS4200           AIX_APPCM          AIX_APPCM,AIX_fcparray
DS4300           AIX_APPCM          AIX_APPCM,AIX_fcparray
DS4500           AIX_APPCM          AIX_APPCM,AIX_fcparray
DS4700           AIX_APPCM          AIX_APPCM,AIX_fcparray
DS4800           AIX_APPCM          AIX_APPCM,AIX_fcparray
DS3950           AIX_APPCM          AIX_APPCM
DS5020           AIX_APPCM          AIX_APPCM
DS5100           AIX_APPCM          AIX_APPCM
DS3500           AIX_APPCM          AIX_APPCM

# mpio_get_config -Av
Frame id 0:
    Storage Subsystem worldwide name: 60ab80016a13700005127aff6
    Controller count: 2
    Partition count: 2
    Partition 0:
    Storage Subsystem Name = ''
        hdisk      LUN #   Ownership          User Label
        hdisk8         4   A (preferred)      testdrive
    Partition 1:
        hdisk      LUN #   Ownership          User Label
        hdisk9         4   B (non-preferred)  testdrive


# lsattr -El hdisk8 | grep fail
algorithm       fail_over                                               Algorithm                        True

# lsattr -El hdisk8 | grep  reserve_policy
reserve_policy  single_path                                             Reserve Policy                   True


dac0 Available 0A-08-02 DS3/4K PCM User Interface
dac1 Available 0A-09-02 DS3/4K PCM User Interface
hdisk0 Available 09-08-00-3,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 09-08-00-4,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 09-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk3 Available 09-08-00-8,0 16 Bit LVD SCSI Disk Drive
hdisk4 Available 09-08-01-3,0 16 Bit LVD SCSI Disk Drive
hdisk5 Available 09-08-01-4,0 16 Bit LVD SCSI Disk Drive
hdisk6 Available 09-08-01-5,0 16 Bit LVD SCSI Disk Drive
hdisk7 Available 09-08-01-8,0 16 Bit LVD SCSI Disk Drive
hdisk8 Available 0A-08-02 MPIO Other DS4K Array Disk
hdisk9 Available 0A-09-02 MPIO Other DS4K Array Disk

As I explained earlier
Fiber Port 0 is connected to SAN Controller A
Fiber Port 1 is connected to SAN Controller B
There is no SAN Switch in between
and it is direct connection

If both Fiber ports are connected to both controllers then I see duplicated disks.

Do I need to download drivers for IBM FastT 1722 - 60U and it should be appear as 1722-600 (600) Disk Array Device ?

or something else need to be done...

Appreciate your help and feedback...

How did you define the WWPN on storage? and how did you map the WWPNs to host.?

There are two ways using IBM Storage CLient Software

1) Use the default group, which has the WWPN hosts direct connection, so i didn't define...

or

2) Under the default group --> Create a new Group/Host --> it will ask you for Host mapping ...

attachment included...

Actually that you see the same disk from different angles is not a bug, it is a feature. If one of these connections (and hence the related hdisk device) would be disconnected the whole system would still work.

This is what "multipath" drivers are all about: to create single devices back from the various device entries, which represent different views of the same hardware.

Not sure what your question is, but i hope this helps.

bakunin

The standard practice would be to have one disk appearing in AIX but with multiple paths.

So hdisk8 should be having two paths : ie., through Controller A an Controller B instead of appearing as Hdisk8 through Controller A and Hdisk9 through Controller B ; To achieve this do I need to place a SAN Switch between AIX machine and SAN Storage and do zoning or is the problem related to config ?

thanks

I know exactly your configuration, since they are directly plugged into each controller, they are seeing it as a single path (even though if you are mapping two WWNs to a single host).

Two options,

  1. 1st install MPIO on this system (if you have on VIO, it won't help as you have a direct connection to SAN from the LPAR itself). I am less confident this might help you resolve the problem.

  2. Add a fabric, and do the zoning, and then map the LUN.

With the way it is seen now, there are high chances of data corruption.

1 Like

Thanks for the reply. I already have MPIO drivers installed in the VIOS. I will try with option 2 and post results. But with option #2 , I need to configure zoning.. I am not familiar with zoning, is it easy ?

Ok,

Zoning to a SAN admin is easy, but if you are new it might take a few efforts to set it. I am not sure what switch you have or (will use).

The SAN Switch I just bought a used one is IBM 2005 H16

In the zone administration of it , default ip http://10.1.1.70 admin/password ; I can see three options Alias , Zone , Zone Config.

Can someone help me find a document for zoning ?

this is the config it came with


The Effective Configuration is:  productionCfg

The Member(s) of the Effective Configuration is (are):

Zone:                 DR_TEMP_ZONE        20:01:00:0e:11:14:82:8f
                                          20:04:00:0e:11:14:82:8f
                                          20:00:00:24:ff:07:1b:94
                                          
Zone:                 dcbackup02prd_zone_0        21:00:00:e0:8b:9b:91:c2
                                                  21:00:00:1b:32:05:7e:f8
                                                  50:05:07:63:0f:42:57:01
                                                  50:05:07:63:0f:42:57:02
                                                  50:05:07:63:0f:42:57:03
                                                  50:05:07:63:0f:42:57:04
                                                  50:05:07:63:0f:42:57:05
                                                  50:05:07:63:0f:42:57:06
                                                  50:05:07:63:0f:42:57:07
                                                  
Zone:                 dcbackup02prd_zone_1        21:00:00:e0:8b:9b:c2:bd
                                                  21:00:00:1b:32:05:7f:f8
                                                  50:05:07:63:0f:42:57:08
                                                  50:05:07:63:0f:42:57:09
                                                  50:05:07:63:0f:42:57:0a
                                                  

Currently effective configuration on the Fabric:  productionCfg

Zone DB Max size for the fabric - 1048576 bytes

Zone DB to be committed - 0 bytes

You have defined the following zoning database:

Aliases:
        DRBACKUP01PRD:
              20:00:00:24:ff:07:1b:94

        DR_TAPE1:
              Bdt B�ro- Und Datentechnik GmbH & Co. Kg 20:01:00:0e:11:14:82:8f

        DR_TAPE2:
              Bdt B�ro- Und Datentechnik GmbH & Co. Kg 20:04:00:0e:11:14:82:8f

        backup520_fcs2_10:
              Emulex Corporation 10:00:00:00:c9:46:4c:cb

        backup520_fcs2_20:
              Emulex Corporation 20:00:00:00:c9:46:4c:cb

        dcbackup02prd_0:
              Qlogic Corp. 21:00:00:e0:8b:9b:91:c2

        dcbackup02prd_1:
              Qlogic Corp. 21:00:00:e0:8b:9b:c2:bd

        dcbackup05prd:
              21:00:00:1b:32:05:7e:f8

        dcbackup05prd_fcs2:
              21:00:00:1b:32:05:7f:f8

        dcbkup01prd_port0:
              21:00:00:1b:32:0e:ec:21

        nvrback4_hba_slot2_20:
              Qlogic Corp. 20:00:00:e0:8b:13:79:80

        nvrback4_hba_slot2_21:
              Qlogic Corp. 21:00:00:e0:8b:13:79:80

        tape0_02:
              IBM 50:05:07:63:0f:02:57:01

        tape0_42:
              IBM 50:05:07:63:0f:42:57:01

        tape1_02:
              IBM 50:05:07:63:0f:02:57:02

        tape1_42:
              IBM 50:05:07:63:0f:42:57:02

        tape2_02:
              IBM 50:05:07:63:0f:02:57:03

        tape2_42:
              IBM 50:05:07:63:0f:42:57:03

        tape3_02:
              IBM 50:05:07:63:0f:02:57:04

        tape3_42:
              IBM 50:05:07:63:0f:42:57:04

        tape4_02:
              IBM 50:05:07:63:0f:02:57:05

        tape4_42:
              IBM 50:05:07:63:0f:42:57:05

        tape5_02:
              IBM 50:05:07:63:0f:02:57:06

        tape5_42:
              IBM 50:05:07:63:0f:42:57:06

        tape6_02:
              IBM 50:05:07:63:0f:02:57:07

        tape6_42:
              IBM 50:05:07:63:0f:42:57:07

        tape7_42:
              IBM 50:05:07:63:0f:42:57:08

        tape8_42:
              IBM 50:05:07:63:0f:42:57:09

        tape9_42:
              IBM 50:05:07:63:0f:42:57:0a

Zones:
        DR_TEMP_ZONE:
              DR_TAPE1      DR_TAPE2
              DRBACKUP01PRD

        bak4tapezone:
              nvrback4_hba_slot2_21      tape4_42
              tape5_42

        dcbackup02prd_zone_0:
              dcbackup02prd_0      dcbackup05prd
              tape0_42      tape1_42
              tape2_42      tape3_42
              tape4_42      tape5_42
              tape6_42

        dcbackup02prd_zone_1:
              dcbackup02prd_1      dcbackup05prd_fcs2
              tape7_42      tape8_42
              tape9_42

        dcbkup01prd_zone:
              dcbkup01prd_port0

        p520tapezone:
              backup520_fcs2_10

Configurations:
        productionCfg:
              dcbackup02prd_zone_0      dcbackup02prd_zone_1
              DR_TEMP_ZONE

        x:
              p520tapezone



Ok,
Aliasing is nothing but giving a name to your WWPN's, as you can see there are many WWPNS already configured. If you have to remove a set of WWPN's from a particular system without Alias you have to navigate all through. But, if you define an Alias you can directly go to that system and cross verify the WWPN's and do the operation.

Zoning is mapping those WWPN's to your SAN (using switch port).

The best practice always is to take a backup/snap shot, we call it CLONE here, do the cloning 1st and then proceed with below steps. If things go awry, you just have to reactive the clone and you are back to the point you cloned it.

Now, you will create an Alias (hostname and WWPNS)
Then you zone it (WWPN ot Switch port).
Then you activate it.

1 Like

Hi filosophizer,

I don't think a switch is the answer...

Your config has fallen out of the IBM "supported" / known to work, configuration area.

The DS is simply too old for your OS level - hence the weird results.

IBM System Storage Interoperation Center (SSIC)

AIX 6.1 TL3 was the latest level to officially support the DS4500 and then the DS had to have the latest firmware.

From AIX 6.1 TL4 the DS native (MPIO / RDAC) support changed to only support the newer generation of DS storage.

You might try switching from MPIO/APPCM back to RDAC/fcparray with manage_disk_drivers to see if that will still work - rmdev -Rdl the adapters and make sure the dacs/dars and disks have all gone before bringing them back with a reboot or cfgmgr to bring in the new (old) driver.

Make sure the DS4k5 firmware is the latest, 06.60.xx, before you go any further...

HTH

1 Like

Thanks for the replies, IBMTECH and Dukessd

My company has an old system P4 with the following config

# prtconf | more
System Model: IBM,7029-6C3
Processor Type: PowerPC_POWER4
Number Of Processors: 2
Processor Clock Speed: 1200 MHz
CPU Type: 64-bit
Kernel Type: 32-bit
LPAR Info: 1 NULL
Memory Size: 16384 MB
Good Memory Size: 16384 MB
Firmware Version: IBM,RG040325_d75e04_s
Console Login: enable
Auto Restart: true
Full Core: false

# lsdev -Cc adapter

fcs0      Available 1H-08    FC Adapter
fcs1      Available 1n-08    FC Adapter

lscfg -vl fcs0
  fcs0             U0.1-P1-I5/Q1  FC Adapter

        Part Number.................00P4295
        EC Level....................A
        Serial Number...............1D4230CADC
        Manufacturer................001D
        Feature Code/Marketing ID...5704
        FRU Number..................     00P4297
        Device Specific.(ZM)........3
        Network Address.............10000000C93E6B3E
        ROS Level and ID............02E01871
        Device Specific.(Z0)........2003806D
        Device Specific.(Z1)........00000000
        Device Specific.(Z2)........00000000
        Device Specific.(Z3)........03000909
        Device Specific.(Z4)........FF601231
        Device Specific.(Z5)........02E01871
        Device Specific.(Z6)........06631871
        Device Specific.(Z7)........07631871
        Device Specific.(Z8)........20000000C93E6B3E
        Device Specific.(Z9)........HS1.81X1
        Device Specific.(ZA)........H1D1.81X1
        Device Specific.(ZB)........H2D1.81X1
        Device Specific.(YL)........U0.1-P1-I5/Q1


# oslevel -r

5200-05

which AIX 5.2 ML 05

# lsdev -Cc disk
hdisk0 Available 1S-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 1n-08-02     1722-600 (600) Disk Array Device
hdisk3 Available 1H-08-02     1722-600 (600) Disk Array Device
hdisk4 Available 1n-08-02     1722-600 (600) Disk Array Device
hdisk5 Available 1H-08-02     1722-600 (600) Disk Array Device
hdisk6 Available 1n-08-02     1722-600 (600) Disk Array Device

lspath -H
status  name   parent

Enabled hdisk0 scsi0
Enabled hdisk1 scsi0

lslpp -l | grep mpio
  devices.common.IBM.mpio.rte

lslpp -lc | grep mpio
/usr/lib/objrepos:devices.common.IBM.mpio.rte:5.2.0.50::COMMITTED:F:MPIO Disk Path Control Module:

manage_disk_drivers -l
ksh: manage_disk_drivers:  not found.

///not found due to old version 5.2

mpio_get_config -Av
ksh: mpio_get_config:  not found.

///not found due to old version 5.2


SAN STORAGE DS4300 
PROFILE FOR STORAGE SUBSYSTEM:  (8/23/13 10:43:42 PM)

SUMMARY------------------------------
   Number of controllers: 2
   Number of arrays: 4
   Total number of logical drives (includes an access logical drive): 6 of 1024 used
      Number of standard logical drives: 5
      Number of access logical drives: 1
   Number of drives: 14
   Supported drive types: Fibre (14)
   Total hot spare drives: 4
      Standby: 4
      In use: 0
   Access logical drive: LUN 31 (see Mappings section for details)
   Default host type: AIX (Host type index 6)
   Current configuration
      Firmware version: 06.12.16.00
      NVSRAM version: N1722F600R912V05
   Pending configuration
      Staged firmware download supported?: Yes
      Firmware version: None
      NVSRAM version: None
      Transferred on: None

But this one doesn't have any issues, Controller A is connected to Port 0 of Fiber Card 1 and Controller B is connected to Port 1 of Fiber Card 2 but there is a switch in between them and there is zoning.

I will zone the switch as IBMTECH suggested and try and post the details...

If anyone know how to solve this issue, please post.
thanks

As for your old POWER 4 machine the AIX MPIO driver (and the older DS storage subsystems) only support APPCM (Active / Passive Path Control Module) so make sure your zoning only allows one path per adapter / controller - as if it were direct attach.
If you zone each adapter to both controllers you'll end up with a mess in the ODM and get some really strange problems.
HTH