Removing LVM Volume Group that doesn't exist anymore

Linux and Unix-Like Red Hat
1

Our SAN administrator decided to unpresent then destroy LUN's we were actively using as a volume group (all PV's in said volume group). Now every time I do a pvscan or whatever it complains about I/O errors trying to access those PV's. How do I get it to forget the VG existed completely? vgreduce --removemissing says the VG doesn't exist (presumably since it couldn't activate any LV's) I can't create another VG of that name because it says it does exist.

2

Have you tried deactivating the LVs and VGs which were on the affected PVs?

lvchange -an vg_name/lv_name
vgchange -an vg_name

After that, you have to use lvremove and vgremove to remove the corresponding LVs and VGs before you can re-use the same name for your new VGs/LVs.

3

No-go:

[root@dfletcher ~]# lvs
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 142811791360: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 142811848704: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 530428395520: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 530428452864: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 4096: Input/output error
  LV               VG          Attr   LSize    Origin Snap%  Move Log Copy%  Convert
  lv_domino8_base  vg2_domino8 -wi-ao  780.96G
  lv_domino8_daos  vg2_domino8 -wi-ao  315.00G
  lv_domino8_tlogs vg2_domino8 -wi-ao   65.00G
  lv_dom14_base    vg_domino14 -wi-ao 1023.00G
  lv_dom14_daos    vg_domino14 -wi-ao  562.41G
  lv_dom14_tlogs   vg_domino14 -wi-ao  135.55G
[root@dfletcher ~]# lvchange -an /dev/vg_domino8/lv_domino8_base
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 142811791360: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 142811848704: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 530428395520: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 530428452864: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg_domino8" not found
  Skipping volume group vg_domino8
[root@dfletcher ~]# lvs
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 142811791360: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 142811848704: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_domino8/lv_domino8_tlogs: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 530428395520: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 530428452864: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_domino8/lv_domino8_base: read failed after 0 of 4096 at 4096: Input/output error
  LV               VG          Attr   LSize    Origin Snap%  Move Log Copy%  Convert
  lv_domino8_base  vg2_domino8 -wi-ao  780.96G
  lv_domino8_daos  vg2_domino8 -wi-ao  315.00G
  lv_domino8_tlogs vg2_domino8 -wi-ao   65.00G
  lv_dom14_base    vg_domino14 -wi-ao 1023.00G
  lv_dom14_daos    vg_domino14 -wi-ao  562.41G
  lv_dom14_tlogs   vg_domino14 -wi-ao  135.55G
4

Run the command under strace:

strace -f -o /some/strace/output/file lvs

Then go through the output file and see where it's finding the config data for those volumes and groups.

If that file is a text-based config file, make a copy so you have a backup just in case, and remove the "disappeared" groups and volumes from the original.

5

LVM metadeta is text-based but it doesn't exist in a file anywhere. They're physically stored in the "LVM Head" of each physical volume in the VG. Since there are no PV's anymore I'm guessing the kernel keeps a record of that in memory somewhere.

6

I understand /dev/vg_domino8/lv_domino8_base is still appearing. If so, try to figure out which dm device it corresponds to (from /dev/mapper ) directory. Then, you may take a look at /sys/block/dm-x (x is the corresponding number for the dm device in question) directory and see if there's any way to "turn off" it. I could not find any such thing though.

Have you rebooted the server? I would have done a reboot in this scenario.

1 Like
7

Unfortunately, we can't reboot this server since it's actively serving end users and that would require a change request. But your post did give me an idea that did work: I deleted /dev/vg_domino8 and /dev/mapper/vg_domino* then ran the pvscan/lvs again and did not receive any errors this time around.

Thanks for putting me on the right track.

8

Great! :b: