Stale PPs in AIX, failed disks.. how to replace?

c3rb3rus · October 27, 2014, 4:48pm

I have a AIX 7.1 system that has 3 failed disks, 1 in rootvg and 2 in vg_usr1.

Here is the output of lspv.

# lspv
hdisk0          00044d4dfbb11575                    vg_usr1         active
hdisk1          0000150179158027                    vg_usr1         active
hdisk2          00001501791582c2                    vg_usr1         active
hdisk3          000015013a9148db                    vg_usr1         active
hdisk4          00001501791587df                    vg_usr1         active
hdisk5          0000150179158a75                    vg_usr1         active
hdisk6          00001501792ff38b                    vg_usr1         active
hdisk7          00001501792ff659                    vg_usr1         active
hdisk8          000015019dbd4c34                    vg_usr1         active
hdisk9          0000150178bb04a2                    vg_usr1         active
hdisk10         0000150173c3f71f                    rootvg          active
hdisk11         000015015b329219                    rootvg          active
hdisk12         0000150115562195                    rootvg          active
hdisk13         0000150173c3f833                    rootvg          active
hdisk14         0000150173c3f860                    rootvg          active
hdisk15         00001501f260dea1                    rootvg          active
hdisk16         00001501791a37d9                    vg_usr2         active
hdisk17         00001501791a3a9d                    vg_usr2         active
hdisk18         00001501791a3d45                    vg_usr2         active
hdisk19         00001501791a3fea                    vg_usr2         active
hdisk20         00001501791a4288                    vg_usr3         active
hdisk21         00001501791a451e                    vg_usr3         active
hdisk22         000015019dbd4ed5                    vg_usr1         active
hdisk23         000015019dbd517b                    vg_paging       active
hdisk24         00001501791bf1c6                    vg_usr1         active
hdisk25         00001501791bf48f                    vg_usr1         active
hdisk26         00001501791bf72e                    vg_usr1         active
hdisk27         00001501791bfa40                    vg_usr1         active
hdisk28         00001501791bfce5                    vg_usr1         active
hdisk29         00001501791bff84                    vg_usr1         active
hdisk30         00001501792ff909                    vg_usr1         active
hdisk32         00001501caf49d70                    vg_usr1         active
hdisk33         0000150179310ec5                    vg_usr1         active
hdisk34         000015010dd840e2                    vg_usr1         active
hdisk35         00001501caf588da                    vg_paging       active
hdisk36         00001501793177ff                    vg_usr3         active
hdisk37         0000150179317ac0                    vg_usr3         active
hdisk38         0000150179317d6d                    vg_usr3         active
hdisk39         0000150179318031                    vg_usr3         active
hdisk40         000015019dc3fe46                    vg_usr3         active
hdisk42         00c55310ce0d3117                    None

Output of lsvg for rootvg and vg_usr1.

# lsvg vg_usr1
VOLUME GROUP:       vg_usr1                  VG IDENTIFIER:  000015010000d60000000141138f2577
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      12012 (1537536 megabytes)
MAX LVs:            256                      FREE PPs:       1 (128 megabytes)
LVs:                2                        USED PPs:       12011 (1537408 megabytes)
OPEN LVs:           2                        QUORUM:         12 (Enabled)
TOTAL PVs:          22                       VG DESCRIPTORS: 22
STALE PVs:          2                        STALE PPs:      768
ACTIVE PVs:         20                       AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 128 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no
DISK BLOCK SIZE:    512
# lsvg rootvg
VOLUME GROUP:       rootvg                   VG IDENTIFIER:  000015010000d6000000013569180e7a
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      3276 (419328 megabytes)
MAX LVs:            256                      FREE PPs:       2346 (300288 megabytes)
LVs:                15                       USED PPs:       930 (119040 megabytes)
OPEN LVs:           13                       QUORUM:         4 (Enabled)
TOTAL PVs:          6                        VG DESCRIPTORS: 6
STALE PVs:          1                        STALE PPs:      9
ACTIVE PVs:         5                        AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no
DISK BLOCK SIZE:    512

Output of lsvg -p rootvg and vg_usr1.

# lsvg -p vg_usr1
vg_usr1:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            546         0           00..00..00..00..00
hdisk1            active            546         0           00..00..00..00..00
hdisk2            active            546         0           00..00..00..00..00
hdisk3            missing           546         0           00..00..00..00..00
hdisk4            active            546         0           00..00..00..00..00
hdisk5            active            546         0           00..00..00..00..00
hdisk6            active            546         0           00..00..00..00..00
hdisk7            active            546         0           00..00..00..00..00
hdisk8            active            546         0           00..00..00..00..00
hdisk9            active            546         0           00..00..00..00..00
hdisk22           active            546         0           00..00..00..00..00
hdisk24           active            546         0           00..00..00..00..00
hdisk25           active            546         0           00..00..00..00..00
hdisk26           active            546         0           00..00..00..00..00
hdisk27           active            546         0           00..00..00..00..00
hdisk28           active            546         0           00..00..00..00..00
hdisk29           active            546         0           00..00..00..00..00
hdisk30           active            546         0           00..00..00..00..00
hdisk31           missing           546         0           00..00..00..00..00
hdisk32           active            546         0           00..00..00..00..00
hdisk33           active            546         0           00..00..00..00..00
hdisk34           active            546         1           00..00..00..00..01
# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk10           active            546         497         109..89..81..109..109
hdisk11           active            546         173         110..56..00..00..07
hdisk12           active            546         177         68..00..00..00..109
hdisk13           active            546         487         110..50..109..109..109
hdisk14           missing           546         511         109..89..95..109..109
hdisk15           active            546         501         110..109..64..109..109

When I query each VG, I found that the following PVs are in a missing state.

PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk3            missing           546         0           00..00..00..00..00
hdisk14           missing           546         0           00..00..00..00..00
hdisk31           missing           546         0           00..00..00..00..00

It does look like I have a disk that is unassigned, "hdisk42", so I'd like to put this one in use while I get 2 other disks to replace with.

I was going to try and remove hdisk31 and put hdisk42 in its place..

My notes for removing, swapping the failed disk and adding a new one in for AIX 7.1 are as follows..

However, something is not happy, I can't get passed the first step.

# unmirrorvg -c 1 vg_usr1 hdisk31
0516-1155 lreducelv: Last good copy of a partition cannot reside on a missing disk.
Try again after reactivating the disk using chpv and varyonvg.
0516-922 rmlvcopy: Unable to remove logical partition copies from
        logical volume lv_usr1.
0516-1135 unmirrorvg: The unmirror of the volume group failed.
        The volume group is still partially or fully mirrored.

Then I tried varyonvg. no success.

# varyonvg vg_usr1
varyonvg # 0516-934 /usr/sbin/syncvg: Unable to synchronize logical volume lv_usr1.
0516-932 /usr/sbin/syncvg: Unable to synchronize volume group vg_usr1.

Any help is appreciated! Thank you.

bakunin · October 27, 2014, 6:10pm

Your notes are correct, but only for working disks. Because it is not clear how the copies in your VG are set up it might well be that you have already lost data.

A word about that first: as a responsible admin you should never, NEVER let 3 disks become missing! You should act immediately when the first one fails. This usually foreshadows in the "errpt", when a disk issues an increasing number of disk errors (usually hdisk error 3, which is temporary). What happens there is that blocks are becoming bad and are relocated to good sectors. When formatting a disk AIX sets aside a number of such contingency sectors. One by one these are used if a sector becomes bad but at one point they are exhausted and then you usually get a hdisk error type 4, which is permanent. I suggest to check the "errpt" and "errpt -a" output respectively to find out what happened to the disks.

Second: take stock of your data. Find out if you still have a good copy of every LP (logical partition) by generating a map file for each LV. When you mirror a LV two (or even three, depending on the number of mirrors) PPs are representing one LP. Check the map files for all the LVs if there are LPs represented only by PPs from hdisk31 or hdisk3. If so, you have lost data and you will have to restore its contents from backup (you do have a backup, don't you??).

You will need to varyon the VG for that. Alas, it will not work when disks are missing, even if the quorum checking is disabled. Use the "force" option for this, also use the "-r" option to varyon in read-only state and the "-n" to disable synchronisation of stale partitions:

varyonvg -fnr vg_usr1

Now generate the map files for analysis and varyoff again:

lslv -m <LVname> > /path/to/file

What comes now depends on what your analysis results in. In case you have not lost data already you can try immediate removal of the failed disks. Varyon again and remove all the missing disks:

varyonvg -f vg_usr1
reducevg -df vg_usr1 hdisk3
reducevg -df vg_usr1 hdisk31

Now try to settle the system:

synclvodm
varyoffvg vg_usr1
varyonvg vg_usr1

The last one has to work without any "brutal" handiwork: without "force"-options or the like. If this works so far you might try to put hdisk42 to work. Tell us how far you got and i will explain how to do that in a separate post.

I hope this helps.

bakunin

c3rb3rus · October 27, 2014, 7:42pm

Thanks for the reply bakunin.

Going down what you suggested, I ran...

# varyonvg -fnr vg_usr1
0516-1293 varyonvg: Volume group is currently in read-write mode.
        Use varyoffvg before varyon volume group in different mode.

So now I try.. "varyoffvg -s vg_usr1".

According to man varyoffvg, -s puts group in system maintenance mode..

But also no luck..

# varyoffvg -s vg_usr1
0516-012 lvaryoffvg: Logical volume must be closed.  If the logical
        volume contains a filesystem, the umount command will close
        the LV device.
0516-942 varyoffvg: Unable to vary off volume group vg_usr1.

So, keep going.. unmount /usr1 and I get..

umount: 0506-349 Cannot unmount /dev/lv_usr1: The requested resource is busy.

Figured out what resources are using /usr1 via fuser and got rid of them, was able to run varyoffvg -s vg_usr1 fine this time.

Next I ran varyoffvg -s vg_usr1, it worked but varyonvg still complained about read-write mode, so ran varyoffvg without the -s switch, it took it and was able to run varyonvg -fnr vg_usr1 fine.

# varyonvg -fnr vg_usr1
PV Status:      hdisk0  00044d4dfbb11575        PVACTIVE
                hdisk1  0000150179158027        PVACTIVE
                hdisk2  00001501791582c2        PVACTIVE
                hdisk3  000015013a9148db        PVMISSING
                hdisk4  00001501791587df        PVACTIVE
                hdisk5  0000150179158a75        PVACTIVE
                hdisk6  00001501792ff38b        PVACTIVE
                hdisk7  00001501792ff659        PVACTIVE
                hdisk8  000015019dbd4c34        PVACTIVE
                hdisk9  0000150178bb04a2        PVACTIVE
                hdisk22 000015019dbd4ed5        PVACTIVE
                hdisk24 00001501791bf1c6        PVACTIVE
                hdisk25 00001501791bf48f        PVACTIVE
                hdisk26 00001501791bf72e        PVACTIVE
                hdisk27 00001501791bfa40        PVACTIVE
                hdisk28 00001501791bfce5        PVACTIVE
                hdisk29 00001501791bff84        PVACTIVE
                hdisk30 00001501792ff909        PVACTIVE
                hdisk31 00001501792ffbcc        PVMISSING
                hdisk32 00001501caf49d70        PVACTIVE
                hdisk33 0000150179310ec5        PVACTIVE
                hdisk34 000015010dd840e2        PVACTIVE
varyonvg: Volume group vg_usr1 is varied on.

I have attached the "map.txt" file which is the output of lslv -m for lv_usr1.. any pointers on analyzing this file to check and see if data is missing?

Am I looking to see that a LP# is not on both failed disks?

bakunin · October 27, 2014, 9:41pm

Excactly. Your listing looks like this:

LP    PP1  PV1               PP2  PV2               PP3  PV3
0001  0111 hdisk0            0056 hdisk1            
0002  0111 hdisk1            0056 hdisk2            
0003  0111 hdisk2            0056 hdisk3            
0004  0111 hdisk3            0056 hdisk4

What you look for are lines like these:

LP    PP1  PV1               PP2  PV2               PP3  PV3
[...]
nnnn  xxxx hdisk31           yyyy hdisk3
nnnn  xxxx hdisk3            yyyy hdisk31
nnnn  xxxx hdisk3            yyyy hdisk3
nnnn  xxxx hdisk31           yyyy hdisk31

I didn't bother to build it but a little sed/awk/grep-filtering should do the trick nicely.

Any of these would mean that you lost both copies of a certain LP and thus data. In this case you will need to restore the respective LV from backup no matter what. You need to verify of this being the case first, because you will not be able to remove the missing disks with the method i described otherwise.

I hope this helps.

bakunin

c3rb3rus · October 27, 2014, 10:29pm

Thanks Bakunin,

I have followed your instructions and compared my map, didn't see faulty results and proceeded.

for vg_usr1 I now have..

# lsvg vg_usr1
VOLUME GROUP:       vg_usr1                  VG IDENTIFIER:  000015010000d60000000141138f2577
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      10920 (1397760 megabytes)
MAX LVs:            256                      FREE PPs:       1 (128 megabytes)
LVs:                2                        USED PPs:       10919 (1397632 megabytes)
OPEN LVs:           2                        QUORUM:         11 (Enabled)
TOTAL PVs:          20                       VG DESCRIPTORS: 20
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         20                       AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no
DISK BLOCK SIZE:    512

Now I believe hdisk42 is a good disk, anyway to test that? How would I go about adding this disk if the above proves true to vg_usr1 to gain space?

Essentially what we did is remove the bad disk because the partitions on it were available on a mirrored disk?

For tests, I was able to mount /usr1 (which is on vg_usr1) and the data is there.

Thanks, definitely taking notes here

bakunin · October 28, 2014, 8:16am

Good!

You do not need to test that. Adding a disk to a VG means formatting it, so the system will tell you if the disk is faulty. Do the following to add a new disk to a VG:

extendvg <vgname> <hdisk-device>

If everything goes well there is no output at all, otherwise diagnostic messages will appear.

Notice that a physical volume (a "disk" in LVM speak) can only contain 1018 PPs in an ordinary VG. This means that because of your PP size of 128M (see lsvg vg_usr1 output) you can only add disks up to somewhat below 128G in size. If your hdisk42 is bigger than this you need to introduce a "factor". This means the usual limit of 32 PVs each 1018 PPs in size is divided/multiplied by this factor: a factor of 2 means that only 16 PVs can be used but each PV can hold 2036 PPs. You change the factor by:

chvg -t <factor> <vgname>

Not quite. You were able to remove the bad disks because you still had one good copy of every LP. Logical volumes (LVs) consist of "logical partitions" (LP) which are in turn made of "PPs" (physical partitions). PPs are parts of a disk. If you create an unmirrored LV you assign one PP to each LP the LV consists of. Creating a mirrored LV means assigning 2 PPs to every LP and writing data to these two different (disk) locations in parallel.

After adding the new disk you might want to remirror the VG. First find out which LVs are affected. Here is an example:

# lsvg -l myvg
myvg:
LV NAME             TYPE       LPs     PPs    PVs    LV STATE      MOUNT POINT
mirrorlv            jfs2       80      160      2    open/syncd    /some/where
loglv04             jfs2log     1       1       1    open/syncd    N/A
nomirrorlv          jfs2        3       3       1    open/syncd    /else/where

The first LV is mirrored, the second is not and the jfslog LV is not mirrored too. Use the mklvcopy command to create mirror copies of unmirrored LVs. You could use the mirrorvg to do that but only if hdisk42 offers the same or more space as the missing disks together, because you do not have any space reserves in your VG (see "free PPs" in the lsvg output).

I hope this helps.

bakunin

c3rb3rus · October 28, 2014, 10:45am

Tried to extend the volume..

# extendvg vg_usr1 hdisk42
0516-1398 extendvg: The physical volume hdisk42, appears to belong to another volume group. Use the force option to add this physical volume to a volume group.
0516-792 extendvg: Unable to extend volume group.

Interesting error, since hdisk42 shows it is not assigned when I ssue "lspv hdisk42", I get..

Physical volume hdisk42 is not assigned to a volume group.

So next, i tried to force it and it worked.. "extendvg -f vg_usr1 hdisk42"

Then I tried mirrorvg vg_usr1, and I get..

# mirrorvg vg_usr1
0516-1119 mirrorvg: Not enough free physical partitions to satisfy request.
0516-1200 mirrorvg: Failed to mirror the volume group.

This coresponds to what you said, since I had 2 disks fail I don't have enough for this command, so next I try.

Looking at lslv lv_usr1 I have it set for Copies = 2, so I will add 2 for the mklvcopy cmdlet.

However, next I get this error.

# mklvcopy lv_usr1 2 hdisk42
0516-1509 mklvcopy: VGDA corruption: physical partition info for this LV is invalid.
0516-842 mklvcopy: Unable to make logical partition copies for
        logical volume.

The output of lslv_lv_usr1 is:

# lslv lv_usr1
LOGICAL VOLUME:     lv_usr1                VOLUME GROUP:   vg_usr1
LV IDENTIFIER:      000015010000d60000000141138f2577.1 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       closed/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            32512                  PP SIZE:        128 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                6005                   PPs:            12010
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       maximum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /usr1                  LABEL:          /usr1
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO
INFINITE RETRY:     no

# lslv -l lv_usr1
0516-1939 : PV identifier not found in VGDA.

# lsvg vg_usr1
VOLUME GROUP:       vg_usr1                  VG IDENTIFIER:  000015010000d60000000141138f2577
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      11466 (1467648 megabytes)
MAX LVs:            256                      FREE PPs:       547 (70016 megabytes)
LVs:                2                        USED PPs:       10919 (1397632 megabytes)
OPEN LVs:           0                        QUORUM:         11 (Enabled)
TOTAL PVs:          21                       VG DESCRIPTORS: 21
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         21                       AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no
DISK BLOCK SIZE:    512

bakunin · October 28, 2014, 4:15pm

First off, your "inter-policy" is set to "maximum", which is nonsense. "inter-policy=maximum" means to use as many PVs as possible when creating or extending a LV, which results in a "poor mans striping". The basic idea is to distribute I/O over as many disks as possible. This might make sense with very small PP sizes (2-16MB) and many very old disks. With todays disks and todays typical PP sizes it is utter nonsense and only makes administration overly complicated. My suggestion is to get rid of it. Use an inter-policy of "minimum" only.

Next: your VG is 1.5 TB in size and has a laughable amount of 128M in contingency! I suggest you get some more disk space to allow at least for reorganisations, which should take place regularly when you deal with physical disks. Here, like in any area, it doesn't pay off to be greedy and it certainly lowers efficiency when using a resource to its abolute limit. You do not use your processor ressources at a permanent 100%, you do not tune your system to death so that it constantly has zero free RAM and for the same reason you do not fill your disks to absolute capacity - there is no room left to react if something unforeseen comes up. Plus some VG operations need unused space to be undertaken. You should have at least 100-200GB free space once you have created all the filesystems, better even more.

My suggestion is to recreate the VG completely anew: backup your data using the savevg utility, delete the VG (you can use exportvg to remove it completely from the system), then recreate it using the restvg . Select sensible values for the PP size when recreating it, because your system seems to use pretty small disks: you have 22 (now 21) PVs in your VG and a net capacity of 1.5TB, the single average disk is about 70G in size. No disk you can get today has as few as 70GB but with your PP size of 128M you can't use any disk larger than ~128G. On the other hand you cannot use a factor as i suggested above, because even the smallest factor of 2 would limit the VG to 16 PVs which you already exceed. Therefore, when recreating the VG use a PP size of 512MB or even bigger, which will allow you to replace the (probably quite old) disks with new, bigger ones. Be warned that this operation will take some time and you need downtime to do it.

If you can get additional disks to hold the data you could just create a new VG and copy the data from the old one, then delete the old one, this way you save the time for creating the backup and you always have a working copy as a fallback solution. A healthy leayout would be 4 PVs of 500GB each, which gives you 2TB of space, 1.5 TB taken and 500GB as a reserve to draw on. This (or the solution above) is the recommended way to do it.

The next best is to reorganize the existing VG on the existing disks. Some problems described above will stay, but at least the problems internal to the VG could be repaired.

Reorganisation means: remove all the mirrors from all the LVs in the VG. The command is unmirrorvg . This way you get some free space to work with. Then change the LVs using the chlv command to set inter-policy to "minimum". Once you have done this use the reorgvg command to bring the new LV layout into place. Be prepared to wait for quite some time (probably some hours) before this is finished, it will physically relocate the LPs of all the LVs. You can do all this online without downtime, but i suggest to do it in times of least activity or even take a downtime anyway. It will certainly be slowed down by much system stress and users will notice the degraded performance when this operation runs. Finally remirror using the mklvcopy command or - if you can get additional disks to add to the VG - a mirrorvg which does a mklvcopy for all LVs.

If you do a mirrorvg here is a tip: switch off the syncing, because it uses only 1 PP at a time. Do it this way (deferred syncing):

# mirrorvg -s myvg
# syncvg -P <NN> -v myvg

For "NN" set a number 1-32, which is the number of PPs be synced in parallel. Which number is good depends on your RAM, load, PP size and other things, but i suggest 10-20 for PP sizes between 64M and 512M.

I hope this helps.

bakunin

/PS:

This might be because a VGDA (volume group descriptor area) was found on the disk. Maybe it was already in use in this or another system and already held data. In any way, when you forced it into the VG it was cleaned of these data and reformatted, so speculations about its former content are moot at this point.

bakunin

c3rb3rus · October 29, 2014, 10:43am

Thanks for the reply.

I am following your 2nd suggestion as this machine is going to be out its door in about 2 months (its over 5+ years old) and is a DR/testing box so not a production machine, less critical. My goal now is to keep it going for the next 2 months.

My goal is to get /usr1 (vg_usr1) working, removing the failed disks and making sure it is mirrored.

Sorry, perhaps I should have stated this first though your notes I am going to use on the new system we are putting in place.

I changed the policy to minimum, and tried to run reorgvg...

# lsvg vg_usr1
VOLUME GROUP:       vg_usr1                  VG IDENTIFIER:  000015010000d60000000141138f2577
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      11466 (1467648 megabytes)
MAX LVs:            256                      FREE PPs:       547 (70016 megabytes)
LVs:                2                        USED PPs:       10919 (1397632 megabytes)
OPEN LVs:           0                        QUORUM:         11 (Enabled)
TOTAL PVs:          21                       VG DESCRIPTORS: 21
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         21                       AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no
DISK BLOCK SIZE:    512

# lslv lv_usr1
LOGICAL VOLUME:     lv_usr1                VOLUME GROUP:   vg_usr1
LV IDENTIFIER:      000015010000d60000000141138f2577.1 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       closed/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            32512                  PP SIZE:        128 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                6005                   PPs:            12010
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /usr1                  LABEL:          /usr1
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO
INFINITE RETRY:     no

# reorgvg vg_usr1
0516-1939 lmigratepp: PV identifier not found in VGDA.
0516-964 reorgvg: Unable to migrate logical volume lv_usr1.
0516-968 reorgvg: Unable to reorganize volume group.
0516-962 reorgvg: Logical volume loglv00 migrated.

Sounds like one of the pvs (hard drives) could be having an issue that is currently assigned to the vg?

bakunin · October 29, 2014, 4:04pm

OK, point taken.

You may have read over it: i suggested to completely unmirror all the LVs (in the VG) first. From your lslv output i can see that at least "lv_usr1" is still mirrored - and most probably some mirror copies of certain LPs are already gone with the missing disks. I suppose the other LVs in the volume are mirrored too and their mirroring status is uncertain in the same way. Furthermore it makes no sense to reorganize a mirrored volume instead of unmirroring it, reorganize it and at last remirror it again. It simply doubles the work and limits the reorganization options.

No, not necessarily. A possible reason is that some PPs with mirror copies are not where the system expects them to be because the disks which used to provide these PPs are gone. Finally: you replaced two missing disks by one new one and most probably you lost raw diskspace. the VG was filled to capacity at the start and if you further reduced the available raw space you cannot mirror all the LVs any more.

Therefore:

unmirror the VG completely. If you worry about possible data loss take a backup before but unmirror anyway. Do exhaustive fsck s before to make sure the FSes are OK. (You will need downtimes for the fsck because the FSes must be unmounted.)
After this is finished take stock of: how much raw space do you have and how much is used. If the space in your VG is already 50% or more used a complete mirror is not possible and you will have to select certain FSes/LVs which will get mirrors and others which don't. One way to circumvent this is to reduce the size of LVs/FSes to get usage below 50% again. Maybe some filesystems could do with a little less space?

If so, here is how to it: this command will reduce a FS "/some/where" and its underlying LV by 1GB. Notice that you can only assign whole PPs to a LV, therefore shrinking/expanding only takes place in steps of the PP size.

chfs -a "size=-1G" /some/where

now do a reorgvg .
set up mirrors again using the "mirrorvg" command. Notice that you might need to move a few LVs a second time because you have an odd number of PVs and mirrors are created on different disks (see your lslv output). You do this with the "migratepv" command, with which you can move around one or all the PPs of a LV to certain positions on certain PVs.

It might take considerably less time to get a second replacement disk and make the number of available PVs even again.

I hope this helps.

bakunin

c3rb3rus · November 20, 2014, 5:12pm

Hey bakunin

I decided to destroy one of my volume groups (it was used for testing) where we had the multiple disk failures to address rootvg being stale..

I found this article and followed it to fix my (rootvg.

Here are the events..

# lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk10           active            546         497         109..89..81..109..109
hdisk11           active            546         173         110..56..00..00..07
hdisk12           active            546         177         68..00..00..00..109
hdisk13           active            546         487         110..50..109..109..109
hdisk14           missing           546         511         109..89..95..109..109
hdisk15           active            546         501         110..109..64..109..109

# unmirrorvg rootvg hdisk14
0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c <diskname>'
        as root user to clear the boot record and avoid a potential boot
        off an old boot image that may reside on the disk from which this
        logical volume is moved/removed.

0301-108 mkboot: Unable to read file blocks. Return code: -1
0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform
        bosboot of system to reinitialize boot records.  Then, user must modify
        bootlist to just include:  hdisk10.

# rmdev -l hdisk14 -d
hdisk14 deleted

# bootinfo -b
hdisk10

# Do a chdev on hdisk1 to remove the pvid: 
chdev -l hdisk1 -a pv=clear

# extendvg rootvg hdisk20
0516-1254 extendvg: Changing the PVID in the ODM.
0516-1398 extendvg: The physical volume hdisk20, appears to belong to
another volume group. Use the force option to add this physical volume
to a volume group.
0516-792 extendvg: Unable to extend volume group.

# extendvg -f rootvg hdisk20

# mirrorvg rootvg
0516-1804 chvg: The quorum change takes effect immediately.
0516-1126 mirrorvg: rootvg successfully mirrored, user should perform
        bosboot of system to initialize boot records.  Then, user must modify
        bootlist to include:  hdisk11 hdisk10.

# # lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk10           active            546         423         109..34..62..109..109
hdisk11           active            546         152         109..36..00..00..07
hdisk12           active            546         177         68..00..00..00..109
hdisk13           active            546         485         110..50..107..109..109
hdisk20           active            546         546         110..109..109..109..109
hdisk15           active            546         501         110..109..64..109..109

# lsvg -l rootvg
rootvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
hd5                 boot       1       2       2    closed/syncd  N/A
hd6                 paging     4       8       2    open/syncd    N/A
hd8                 jfs2log    1       2       2    open/syncd    N/A
hd4                 jfs2       2       4       2    open/syncd    /
hd2                 jfs2       26      52      2    open/syncd    /usr
hd9var              jfs2       9       18      2    open/syncd    /var
hd3                 jfs2       2       4       2    open/syncd    /tmp
hd1                 jfs2       320     640     2    open/syncd    /home
hd10opt             jfs2       7       14      2    open/syncd    /opt
hd11admin           jfs2       1       2       2    open/syncd    /admin
fwdump              jfs2       8       16      2    open/syncd    /var/adm/ras/platform
lg_dumplv           sysdump    48      96      2    open/syncd    N/A
livedump            jfs2       8       16      2    open/syncd    /var/adm/ras/livedump
paging00            paging     4       8       2    open/syncd    N/A
lv_inst             jfs2       55      110     2    closed/syncd  N/A

# bootinfo -b
hdisk10
# bootlist -m normal -o
hdisk10 blv=hd5 pathid=0
-
# bosboot -a

bosboot: Boot image is 55324 512 byte blocks.
# bootlist -m normal -o
hdisk10 blv=hd5 pathid=0
-

Now I did see something strange.. I have not yet phisically removed the failed disk (hdisk14) so when I run cfgmgr it gets picked up..

# lsdev -Cc disk | grep "hdisk14"
hdisk14 Available 03-08-01-5,0  16 Bit LVD SCSI Disk Drive
# lspv | grep "hdisk14"
hdisk14         none                                None

In turn, if I now run "bootlist -m normal -o" I get..

hdisk10 blv=hd5 pathid=0
hdisk14 pathid=0

Why is it showing there? I am concerned if a reboot happens I may not be able to boot.. how can I remove it and assign hdisk20 which is the replacement for hdisk14 in rootvg... or do I need to do anything further at this point?

Maybe just rmdev -l hdisk14 -d and leave it?

---------- Post updated 11-20-14 at 02:12 PM ---------- Previous update was 11-19-14 at 06:58 PM ----------

I am going to re-create the vg/lv and start from scratch.

bakunin · November 21, 2014, 2:27am

c3rb3rus:

In turn, if I now run "bootlist -m normal -o" I get..
hdisk10 blv=hd5 pathid=0
hdisk14 pathid=0
Why is it showing there? I am concerned if a reboot happens I may not be able to boot.. how can I remove it and assign hdisk20 which is the replacement for hdisk14 in rootvg... or do I need to do anything further at this point?

There are two commands you need to know: "bosboot" and "bootlist".

With "bootlist" you can set (or display) the order (in NVRAM) in which devices are used to boot from. There are basically 2 lists of such boot devices, "normal" and "service". There is a "key position" that determines which boot list is getting used. Back then, in the times of physical machines, this was really a lock with a key and the key could be in one of 3 possible positions, "normal", "service" (sometimes named "maint") and "lock". Today this is only a virtual construct, but the terminology stuck.

With "bosboot" you bring a boot image to boot from onto a device. This copies the code that is executed during the boot onto the device you designated as boot device with "bootlist".

I suggest you read carefully the man pages of both commands, then read them again. You can render the system unusable with these commands, so spend some time and effort to really, firmly understand what they are doing. Only then use them to alter the boot list. (You will need both of them - proceed only if you understand why.)

I hope this helps.

bakunin