issue AIX MPIO path closed

Phat · March 13, 2019, 7:50am

Hello,
I have encountered the issue on my AIX 6.1. As you can see below, the hdisk1 & hdisk8 are not recognized by the system

[root@xxx] / > lspv
hdisk0          00fa6d1288c820aa                    rootvg          active
hdisk1          00fa6d1288c8213c                    None
hdisk2          00c1cc14d6de272b                    vg_3            active
hdisk3          00c1cc14d6de27e5                    vg_3            active
hdisk4          00c1cc14d6de2898                    vg_3            active
hdisk5          00c1cc14d6de2958                    vg_3            active
hdisk6          00c1cc14d6de2a07                    vg_3            active
hdisk7          00c1cc14716beb5c                    vg_3            active
hdisk8          00c1cc14816d37d0                    None
hdisk9          00c1cc14816d38a0                    vg_3            active
hdisk10         00c1cc14816d396c                    vg_3            active
hdisk11         00c1cc14716beb98                    vg_3            active
hdisk12         00c1cc14716bebd9                    vg_3            active

Futher checking on MPIO, I can see that the path(fcs0, fcs1, fcs3) for hdisk1 & hdisk8 are closed while for another hdisks are still working.

[root@xxx] / > lsmpio
name     path_id  status   path_status  parent  connection

hdisk0   0        Enabled  Pas          fscsi0  201600a0b86e87a4,0
hdisk0   1        Enabled  Pas          fscsi1  202600a0b86e87a4,0
hdisk0   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,0
hdisk1   0        Enabled  Clo          fscsi0  201600a0b86e87a4,1000000000000
hdisk1   1        Enabled  Clo          fscsi1  202600a0b86e87a4,1000000000000
hdisk1   3        Enabled  Clo          fscsi3  202700a0b86e87a4,1000000000000
hdisk2   0        Enabled  Pas          fscsi0  201600a0b86e87a4,2000000000000
hdisk2   1        Enabled  Pas          fscsi1  202600a0b86e87a4,2000000000000
hdisk2   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,2000000000000
hdisk3   0        Enabled  Pas          fscsi0  201600a0b86e87a4,3000000000000
hdisk3   1        Enabled  Pas          fscsi1  202600a0b86e87a4,3000000000000
hdisk3   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,3000000000000
hdisk4   0        Enabled  Pas          fscsi0  201600a0b86e87a4,4000000000000
hdisk4   1        Enabled  Pas          fscsi1  202600a0b86e87a4,4000000000000
hdisk4   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,4000000000000
hdisk5   0        Enabled  Pas          fscsi0  201600a0b86e87a4,5000000000000
hdisk5   1        Enabled  Pas          fscsi1  202600a0b86e87a4,5000000000000
hdisk5   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,5000000000000
hdisk6   0        Enabled  Pas          fscsi0  201600a0b86e87a4,6000000000000
hdisk6   1        Enabled  Pas          fscsi1  202600a0b86e87a4,6000000000000
hdisk6   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,6000000000000
hdisk7   0        Enabled  Pas          fscsi0  201600a0b86e87a4,7000000000000
hdisk7   1        Enabled  Pas          fscsi1  202600a0b86e87a4,7000000000000
hdisk7   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,7000000000000
hdisk8   0        Enabled  Clo          fscsi0  201600a0b86e87a4,8000000000000
hdisk8   1        Enabled  Clo          fscsi1  202600a0b86e87a4,8000000000000
hdisk8   2        Enabled  Clo          fscsi3  202700a0b86e87a4,8000000000000
hdisk9   0        Enabled  Sel,Act      fscsi0  201600a0b86e87a4,9000000000000
hdisk9   1        Enabled  Act          fscsi1  202600a0b86e87a4,9000000000000
hdisk9   3        Enabled  Pas          fscsi3  202700a0b86e87a4,9000000000000
hdisk10  0        Enabled  Sel,Act      fscsi0  201600a0b86e87a4,a000000000000
hdisk10  1        Enabled  Act          fscsi1  202600a0b86e87a4,a000000000000
hdisk10  3        Enabled  Pas          fscsi3  202700a0b86e87a4,a000000000000
hdisk11  0        Enabled  Pas          fscsi0  201600a0b86e87a4,b000000000000
hdisk11  1        Enabled  Pas          fscsi1  202600a0b86e87a4,b000000000000
hdisk11  3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,b000000000000
hdisk12  0        Enabled  Pas          fscsi0  201600a0b86e87a4,c000000000000
hdisk12  1        Enabled  Pas          fscsi1  202600a0b86e87a4,c000000000000
hdisk12  3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,c000000000000

Can someone advise how to do and is there a way to manually change the state of the paths?

filosophizer · March 14, 2019, 2:13am

Hello
can you please post the output of

lspath

and

lsdev -Cc disk

Phat · March 14, 2019, 7:07am

Hello,

Here are the output

[root@xxx] / > lspath
Enabled hdisk0  fscsi0
Enabled hdisk1  fscsi0
Enabled hdisk0  fscsi1
Enabled hdisk1  fscsi1
Enabled hdisk8  fscsi3
Enabled hdisk8  fscsi0
Enabled hdisk0  fscsi3
Enabled hdisk1  fscsi3
Enabled hdisk2  fscsi0
Enabled hdisk3  fscsi0
Enabled hdisk4  fscsi0
Enabled hdisk5  fscsi0
Enabled hdisk6  fscsi0
Enabled hdisk7  fscsi0
Enabled hdisk9  fscsi0
Enabled hdisk10 fscsi0
Enabled hdisk11 fscsi0
Enabled hdisk12 fscsi0
Enabled hdisk2  fscsi1
Enabled hdisk3  fscsi1
Enabled hdisk4  fscsi1
Enabled hdisk5  fscsi1
Enabled hdisk6  fscsi1
Enabled hdisk7  fscsi1
Enabled hdisk8  fscsi1
Enabled hdisk9  fscsi1
Enabled hdisk10 fscsi1
Enabled hdisk11 fscsi1
Enabled hdisk12 fscsi1
Enabled hdisk2  fscsi3
Enabled hdisk3  fscsi3
Enabled hdisk4  fscsi3
Enabled hdisk5  fscsi3
Enabled hdisk6  fscsi3
Enabled hdisk7  fscsi3
Enabled hdisk9  fscsi3
Enabled hdisk10 fscsi3
Enabled hdisk11 fscsi3
Enabled hdisk12 fscsi3


[root@xxx] / > lsdev -Cc disk
hdisk0  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk1  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk2  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk3  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk4  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk5  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk6  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk7  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk8  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk9  Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk10 Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk11 Available 23-T1-01 MPIO DS5100/5300 Disk
hdisk12 Available 23-T1-01 MPIO DS5100/5300 Disk

filosophizer · March 14, 2019, 11:02am

HDISK1 and HDISK8 are empty disks. Create a volume group on it and check it out.

Phat · March 15, 2019, 12:28am

Hello,

After creating VG and recheck, I can see that everything is fine for now.

[root@xxx] / > lspv
hdisk0          00fa6d1288c820aa                    rootvg          active
hdisk1          00fa6d1288c8213c                    vg_pt           active
hdisk2          00c1cc14d6de272b                    vg_3            active
hdisk3          00c1cc14d6de27e5                    vg_3            active
hdisk4          00c1cc14d6de2898                    vg_3            active
hdisk5          00c1cc14d6de2958                    vg_3            active
hdisk6          00c1cc14d6de2a07                    vg_3            active
hdisk7          00c1cc14716beb5c                    vg_3            active
hdisk8          00c1cc14816d37d0                    vg_pt           active
hdisk9          00c1cc14816d38a0                    vg_3            active
hdisk10         00c1cc14816d396c                    vg_3            active
hdisk11         00c1cc14716beb98                    vg_3            active
hdisk12         00c1cc14716bebd9                    vg_3            active

And the path is also ok now.

[root@xxx] / > lsmpio
name     path_id  status   path_status  parent  connection

hdisk0   0        Enabled  Pas          fscsi0  201600a0b86e87a4,0
hdisk0   1        Enabled  Pas          fscsi1  202600a0b86e87a4,0
hdisk0   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,0
hdisk1   0        Enabled  Pas          fscsi0  201600a0b86e87a4,1000000000000
hdisk1   1        Enabled  Pas          fscsi1  202600a0b86e87a4,1000000000000
hdisk1   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,1000000000000
hdisk2   0        Enabled  Pas          fscsi0  201600a0b86e87a4,2000000000000
hdisk2   1        Enabled  Pas          fscsi1  202600a0b86e87a4,2000000000000
hdisk2   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,2000000000000
hdisk3   0        Enabled  Pas          fscsi0  201600a0b86e87a4,3000000000000
hdisk3   1        Enabled  Pas          fscsi1  202600a0b86e87a4,3000000000000
hdisk3   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,3000000000000
hdisk4   0        Enabled  Pas          fscsi0  201600a0b86e87a4,4000000000000
hdisk4   1        Enabled  Pas          fscsi1  202600a0b86e87a4,4000000000000
hdisk4   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,4000000000000
hdisk5   0        Enabled  Pas          fscsi0  201600a0b86e87a4,5000000000000
hdisk5   1        Enabled  Pas          fscsi1  202600a0b86e87a4,5000000000000
hdisk5   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,5000000000000
hdisk6   0        Enabled  Pas          fscsi0  201600a0b86e87a4,6000000000000
hdisk6   1        Enabled  Pas          fscsi1  202600a0b86e87a4,6000000000000
hdisk6   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,6000000000000
hdisk7   0        Enabled  Pas          fscsi0  201600a0b86e87a4,7000000000000
hdisk7   1        Enabled  Pas          fscsi1  202600a0b86e87a4,7000000000000
hdisk7   3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,7000000000000
hdisk8   0        Enabled  Sel,Act      fscsi0  201600a0b86e87a4,8000000000000
hdisk8   1        Enabled  Act          fscsi1  202600a0b86e87a4,8000000000000
hdisk8   2        Enabled  Pas          fscsi3  202700a0b86e87a4,8000000000000
hdisk9   0        Enabled  Sel,Act      fscsi0  201600a0b86e87a4,9000000000000
hdisk9   1        Enabled  Act          fscsi1  202600a0b86e87a4,9000000000000
hdisk9   3        Enabled  Pas          fscsi3  202700a0b86e87a4,9000000000000
hdisk10  0        Enabled  Sel,Act      fscsi0  201600a0b86e87a4,a000000000000
hdisk10  1        Enabled  Act          fscsi1  202600a0b86e87a4,a000000000000
hdisk10  3        Enabled  Pas          fscsi3  202700a0b86e87a4,a000000000000
hdisk11  0        Enabled  Pas          fscsi0  201600a0b86e87a4,b000000000000
hdisk11  1        Enabled  Pas          fscsi1  202600a0b86e87a4,b000000000000
hdisk11  3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,b000000000000
hdisk12  0        Enabled  Pas          fscsi0  201600a0b86e87a4,c000000000000
hdisk12  1        Enabled  Pas          fscsi1  202600a0b86e87a4,c000000000000
hdisk12  3        Enabled  Sel,Act      fscsi3  202700a0b86e87a4,c000000000000

How can you know they are empty disks?As we can think the system is using it as non-LVM disks? And second question is, the multipath should also work for them right? Why after adding to LVM, the multipath starts working?

Phat · March 15, 2019, 2:17am

1 more question, can we change the path status: clo, sel, act, ... or it is done by mpio default?

Phat · March 17, 2019, 11:53pm

Hello filosophizer,

can you share me the info still waiting ...

M_Nixon · May 30, 2019, 8:28am

sudo readvgda hdisk0

empty disks will show something like

Invalid lvm_rec!
00000E00   00000000 00000000 00000000 00000000  |................|
00000E10   00000000 00000000 00000000 00000000  |................|
00000E20   00000000 00000000 00000000 00000000  |................|
00000E30   00000000 00000000 00000000 00000000  |................|
00000E40   00000000 00000000 00000000 00000000  |................|
00000E50   00000000 00000000 00000000 00000000  |................|
00000E60   00000000 00000000 00000000 00000000  |................|
00000E70   00000000 00000000 00000000 00000000  |................|
00000E80   00000000 00000000 00000000 00000000  |................|
00000E90   00000000 00000000 00000000 00000000  |................|
00000EA0   00000000 00000000 00000000 00000000  |................|
00000EB0   00000000 00000000 00000000 00000000  |................|
00000EC0   00000000 00000000 00000000 00000000  |................|
00000ED0   00000000 00000000 00000000 00000000  |................|
00000EE0   00000000 00000000 00000000 00000000  |................|
00000EF0   00000000 00000000 00000000 00000000  |................|
read_lvm_rec had an error, exiting

Disks in use will show LV/RAC/VX type info.

Phat · May 31, 2019, 3:37am

It's helpful

bakunin · June 14, 2019, 2:39am

phat:

[root@xxx] / > lspv
hdisk0          00fa6d1288c820aa                    rootvg          active
hdisk1          00fa6d1288c8213c                    None
hdisk2          00c1cc14d6de272b                    vg_3            active
hdisk3          00c1cc14d6de27e5                    vg_3            active
hdisk4          00c1cc14d6de2898                    vg_3            active
hdisk5          00c1cc14d6de2958                    vg_3            active
hdisk6          00c1cc14d6de2a07                    vg_3            active
hdisk7          00c1cc14716beb5c                    vg_3            active
hdisk8          00c1cc14816d37d0                    None
hdisk9          00c1cc14816d38a0                    vg_3            active
hdisk10         00c1cc14816d396c                    vg_3            active
hdisk11         00c1cc14716beb98                    vg_3            active
hdisk12         00c1cc14716bebd9                    vg_3            active

Look at the quoted output above: the lspv commands lists not only the disks but also to which VG they belong to. i.e. hdisk7, hdisk8, hdisk9, etc. all belong to the volume group vg_3 . The two remaining disks, hdisk1 and hdisk8, belong to no VG, hence "None".

There are two possible scenarios: the disks contain a VG but where brought newly to this system and the system doesn't know about the VG. You can use the readvgda command in this case to look if the disk is empty or not and - if it is not - do a "learning import":

importvg -y <VGname> [-V <MajorNumber>] [-n] <hdisk-device>

The second scenario is that the disks are really empty. In this case you just create a new VG or add them to an existing one.

You can tell the difference by the readvgda -command as described above which reads the "volume group descriptor area" of a disk. But you can also do a

chdev -a pv=yes <hd-device>

which either reads an already existing PV-ID (physical volume ID) and adds it to the ODM or, if none is there, creates one on the disk. Then you can try the above-mentioned importvg -command which will either fail (if the disk is empty) or succeed.

I hope this helps.

bakunin