Unable to remove VIOS disk

Hello,

I am unable to remove the disk, whenever i remove the disk using

rmdev -dl hdisk2

or

rmdev -Rdl hdisk2

the disk appears back when i run

cfgmgr

but unable to create any volume group on it

# mkvg -y foovg hdisk2
0516-008 /usr/sbin/mkvg: LVM system call returned an unknown
        error code (-267).
0516-1184 /usr/sbin/mkvg: IO failure on hdisk2.
0516-862 /usr/sbin/mkvg: Unable to create volume group.

but shows the disk is available

#lspv
hdisk9          00c7780e5a93d490                    rootvg          active

hdisk2          none                                None

and the errpt shows



root@:/>errpt -a | more

DISK DRIVE
DISK DRIVE ELECTRONICS

        Recommended Actions
        PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
PATH ID
           0
SENSE DATA
0A00 2800 0000 0000 0000 0104 0000 0000 0000 0000 0000 0000 0102 0000 7000 0400
0000 000A 0000 0000 3E01 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0200 0000 0000 0000 0000 0000 0000 0000 0083 0000
0000 0027 0017
---------------------------------------------------------------------------
LABEL:          SC_DISK_ERR2
IDENTIFIER:     B6267342

Date/Time:       Tue Nov  4 16:17:28 SAUST 2014

# lsdev -Cc disk

hdisk2  Available  Virtual SCSI Disk Drive
hdisk3  Defined    Virtual SCSI Disk Drive

how can i remove this bad disk ?

Hi,

What is the output of;

# lspv
# chdev -l hdisk2 -a pv=clear
# chdev -l hdisk2 -a pv=yes
# lspv

If you see the status change it's an indication that you can at least access the disk correctly. This might go back to the IBM write and verify problem.

Regards

Dave

It doesn't work.
actually the disk doesn't exist, but some how it is showing up in the LPAR


root@clodb:/>chdev -l hdisk2 -a pv=clear
Method error (/etc/methods/chgdisk):
        0514-047 Cannot access a device.
     pv

root@clodb:/>chdev -l hdisk2 -a pv=yes
Method error (/etc/methods/chgdisk):
        0514-047 Cannot access a device.
     pv

root@clodb:/>lspv
hdisk9          00c7780e5a93d490                    rootvg          active
hdisk10         00c7780e9a335af3                    backupvg        active
hdisk13         00c7780eb79e72f6                    oradbvg         active
hdisk11         00c7780e723bb1e0                    bkclodbvg       active
hdisk2          none                                None
root@clodb:/>

I did ODM delete also, restarted the machines then the hdisk appeared as hdisk0

but it is not going.

Hi,

I think that you have an exclusive lock, either held by an other LPAR or VIO - you may want to check with the SAN team that the Zoneing is correct and it hasn't been zoned to an other server/vio.

Regards

Dave

Yes, you are right, it was coming from SAN to VIO Server and from VIO server to LPAR

Now the SAN Connection has been removed from the PSeries Machine

and I tried to remove the DISK from VIO Server but couldn't

# lspv
hdisk0          00c7780e79838606                    rootvg          active
hdisk8          00c7780e8945b5bb                    patchtest       active
hdisk9          00c7780e8945b5bb                    patchtest       active

# lsdev -Cc disk
hdisk0 Available 09-08-00-3,0 16 Bit LVD SCSI Disk Drive
hdisk8 Available 0A-09-02     MPIO Other DS4K Array Disk
hdisk9 Available 0A-09-02     MPIO Other DS4K Array Disk



# varyoffvg patchtest
0516-062 lqueryvg: Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics
0516-012 lvaryoffvg: Logical volume must be closed.  If the logical
        volume contains a filesystem, the umount command will close
        the LV device.
0516-942 varyoffvg: Unable to vary off volume group patchtest.


How can I remove the volume group and the disks ? thanks

I am confused as to how the disk is coming to client?
Have you created a VG on VIOS, created LV and giving those LV's as vscsi disks to client? or you have mapped the whole disk to client?
If your answer is latter then you cannot create a VG on VIOS.

Can you provide the below info

On VIOS (as padmin)
lsdev -slots
lsmap -vadapter vscsiX -all   X=client ID
lspv -free

Now as root
lsvg -l patchtest
lsvg -p patchtest

On client run this

lsvg 
lsvg -o
lspv
lsdev -Cc disk
df -g

The Disk were created on SAN then it were mapped to VIO Server
From VIO server it was mapped to the LPAR

The disk in question on VIO Server is hdisk8 and hdisk9 with the volume group called patchtest. I was able to remove hdisk8 with rmdev -dl command.
but I could not remove hdisk9

$ lsdev -slots
# Slot                    Description       Device(s)
U787B.001.DNW3313-P1-C1   Logical I/O Slot  pci10 fcs0 fcs1
U787B.001.DNW3313-P1-C2   Logical I/O Slot  pci11 sisscsia1
U787B.001.DNW3313-P1-C3   Logical I/O Slot  pci4 pci5 lai0
U787B.001.DNW3313-P1-C4   Logical I/O Slot  pci6 sisioa0
U787B.001.DNW3313-P1-C5   Logical I/O Slot  pci7
U787B.001.DNW3313-P1-T7   Logical I/O Slot  pci2 usbhc0 usbhc1 usbhc2
U787B.001.DNW3313-P1-T9   Logical I/O Slot  pci8 ent0 ent1
U787B.001.DNW3313-P1-T14  Logical I/O Slot  pci9 sisscsia0
U787B.001.DNW3313-P1-T16  Logical I/O Slot  pci3 ide0
U9113.550.107780E-V1-C2   Virtual I/O Slot  ibmvmc0
U9113.550.107780E-V1-C3   Virtual I/O Slot  ent2
U9113.550.107780E-V1-C4   Virtual I/O Slot  ent3
U9113.550.107780E-V1-C5   Virtual I/O Slot  ent4
U9113.550.107780E-V1-C6   Virtual I/O Slot  ent5
U9113.550.107780E-V1-C10  Virtual I/O Slot  vts0
U9113.550.107780E-V1-C11  Virtual I/O Slot  vhost0
U9113.550.107780E-V1-C12  Virtual I/O Slot  vts1
U9113.550.107780E-V1-C13  Virtual I/O Slot  vhost1
U9113.550.107780E-V1-C14  Virtual I/O Slot  vts2
U9113.550.107780E-V1-C15  Virtual I/O Slot  vhost2
U9113.550.107780E-V1-C16  Virtual I/O Slot  vts3
U9113.550.107780E-V1-C17  Virtual I/O Slot  vhost3
U9113.550.107780E-V1-C18  Virtual I/O Slot  vts4
U9113.550.107780E-V1-C19  Virtual I/O Slot  vhost4
U9113.550.107780E-V1-C20  Virtual I/O Slot  vts5
U9113.550.107780E-V1-C21  Virtual I/O Slot  vhost5
U9113.550.107780E-V1-C22  Virtual I/O Slot  vts6
U9113.550.107780E-V1-C23  Virtual I/O Slot  vhost6
U9113.550.107780E-V1-C24  Virtual I/O Slot  vts7
U9113.550.107780E-V1-C25  Virtual I/O Slot  vhost7
U9113.550.107780E-V1-C26  Virtual I/O Slot  vts8
U9113.550.107780E-V1-C27  Virtual I/O Slot  vhost8
U9113.550.107780E-V1-C28  Virtual I/O Slot  vts9
U9113.550.107780E-V1-C29  Virtual I/O Slot  vhost9



$ lsmap -all
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
..........................................................................................

SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost2          U9113.550.107780E-V1-C15                     0x00000004

VTD                   vtopt0
Status                Available
LUN                   0x8100000000000000
Backing device
Physloc

VTD                   vtscsi0
Status                Available
LUN                   0x8500000000000000
Backing device        clodba
Physloc

VTD                   vtscsi5
Status                Available
LUN                   0x8200000000000000
Backing device        rootvg_61_2
Physloc

VTD                   vtscsi14
Status                Available
LUN                   0x8600000000000000
Backing device        bkclodb
Physloc

VTD                   vtscsi30
Status                Available
LUN                   0x8300000000000000
Backing device        test_compress
Physloc

VTD                   vtscsi31
Status                Available
LUN                   0x8400000000000000
Backing device        testpatch
Physloc

SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost3          U9113.550.107780E-V1-C17                     0x00000005


$ lspv -free
NAME            PVID                                SIZE(megabytes)
hdisk7          00c7780e5293914b                    286102

$ oem_setup_env
# lsvg -l patchtest
0516-062 : Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics
# lsvg -p patchtest
0516-062 : Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics

on the client, the disk which has problem is hdisk0


root@clodb:/>lspv
hdisk9          00c7780e5a93d490                    rootvg          active
hdisk10         00c7780e9a335af3                    backupvg        active
hdisk13         00c7780eb79e72f6                    oradbvg         active
hdisk11         00c7780e723bb1e0                    bkclodbvg       active
hdisk0          none                                None
root@clodb:/>lsvg
rootvg
backupvg
oradbvg
bkclodbvg
root@clodb:/>lsvg -o
oradbvg
backupvg
bkclodbvg
rootvg
root@clodb:/>lsdev -Cc disk
hdisk0  Available  Virtual SCSI Disk Drive
hdisk3  Defined    Virtual SCSI Disk Drive
hdisk4  Defined    Virtual SCSI Disk Drive
hdisk5  Defined    Virtual SCSI Disk Drive
hdisk6  Defined    Virtual SCSI Disk Drive
hdisk7  Defined    Virtual SCSI Disk Drive
hdisk8  Defined    Virtual SCSI Disk Drive
hdisk9  Available  Virtual SCSI Disk Drive
hdisk10 Available  Virtual SCSI Disk Drive
hdisk11 Available  Virtual SCSI Disk Drive
hdisk12 Defined    Virtual SCSI Disk Drive
hdisk13 Available  Virtual SCSI Disk Drive
root@clodb:/>df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           1.50      1.26   17%     2262     1% /
/dev/hd2           3.00      0.98   68%    34083    13% /usr
/dev/hd9var        1.00      0.67   34%     4650     3% /var
/dev/hd3           1.00      0.99    1%       61     1% /tmp
/dev/hd1           1.00      0.77   23%       76     1% /home
/proc                 -         -    -         -     -  /proc
/dev/hd10opt       0.50      0.35   30%     3710     5% /opt
/dev/livedump      0.50      0.50    1%        4     1% /var/adm/ras/livedump
/dev/fslv03       19.00     16.85   12%        6     1% /backup
/dev/fslv00      130.00      4.20   97%      421     1% /oradata
/dev/fslv01       25.00      1.70   94%       29     1% /oradata2
/dev/fslv02       14.00      2.39   83%   133321    20% /oratech
/dev/fslv04       24.00      1.47   94%        7     1% /bkclodb
root@clodb:/>

How can I get it removed from VIO ?

Ok, so LV is given as disk to client.
Do this
On VIOS (as root)

lsvg -o
lsvg -l <vgname>

In which VG do you find the LV testpatch?

Now go to client and do
rmdev -Rdl hdisk0

Now go to VIOS (run as padmin)
rmvdev -vtd vtscsi31
The above command will remove the mapping from vhost2 for that LV.

If you run cfgmgr on client you won't find the hdisk0 now.

Now remove the LV testpatch from VIOS
rmlv -f testpatch
If the VG has NO more LVs mapped to any other partition then it will varyoff, if not then you varyoffvg the VG

varyoffvg <vgname>
exportvg <vgname>
rmdev -Rdl hdisk9
1 Like

okay, thanks
on the client side after removing the hdisk0 it is gone, however on the VIOS side

$ rmvdev -vtd vtscsi31

$ rmlv -f testpatch
*******************************************************************************
The command's response was not recognized.  This may or may not indicate a problem.
*******************************************************************************
*******************************************************************************
The command's response was not recognized.  This may or may not indicate a problem.
*******************************************************************************
rmlv: Unable to remove logical volume testpatch.

$ oem_setup_env

# rmlv -f testpatch
0516-062 lquerylv: Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics
0516-062 lqueryvg: Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics
0516-912 rmlv: Unable to remove logical volume testpatch.
# varyoffvg testpatch
0516-306 getlvodm: Unable to find volume group testpatch in the Device
        Configuration Database.
0516-942 varyoffvg: Unable to vary off volume group testpatch.
# lspv
hdisk0          00c7780e79838606                    rootvg          active
hdisk1          00c7780e2e21ec86                    diskpool_4      active
hdisk2          00c7780ea5bd16bb                    diskpool_4      active
hdisk3          00c7780ee224f286                    disk_pool_5     active
hdisk4          00c7780e1b75933b                    diskpool_3      active
hdisk5          00c7780ece91bde2                    diskpool_2      active
hdisk6          00c7780ec2b65f4d                    diskpool_1      active
hdisk7          00c7780e5293914b                    None
hdisk9          00c7780e8945b5bb                    patchtest       active


unable to remove it VG = patchtest

Ok, what is the output of
lsvg -l patchtest

It is corrupt or damaged
the output is

# lsvg -l patchtest
0516-062 : Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics

I feel there is alteast one more LV (may be more) that is still assigned to client(s).
Ok , do this
On VIOS as root

lsfs
lsvg -l `lsvg`

Compare the output of those two, see which LV is missing from "lsvg -l 'lsvg`" output.
Look for that LV and see if that is assigned as backing device to any other client.

ok

# lsfs
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/hd4        --         /                      jfs2  1048576 --         yes  no
/dev/hd1        --         /home                  jfs2  20971520 --         yes  no
/dev/hd2        --         /usr                   jfs2  7340032 --         yes  no
/dev/hd9var     --         /var                   jfs2  2097152 --         yes  no
/dev/hd3        --         /tmp                   jfs2  7340032 --         yes  no
/dev/hd11admin  --         /admin                 jfs2  1048576 --         yes  no
/proc           --         /proc                  procfs --      --         yes  no
/dev/hd10opt    --         /opt                   jfs2  3145728 --         yes  no
/dev/livedump   --         /var/adm/ras/livedump  jfs2  1048576 --         yes  no
/dev/fwdump     --         /var/adm/ras/platform  jfs2  1048576 --         no   no
/dev/VMLibrary  --         /var/vio/VMLibrary     jfs2  31457280 rw         yes  no
/dev/fslv00     --         /space                 jfs2  --      rw         no   no

and

# lsvg -l `lsvg`
rootvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
hd5                 boot       1       1       1    closed/syncd  N/A
hd6                 paging     1       1       1    open/syncd    N/A
paging00            paging     2       2       1    open/syncd    N/A
hd8                 jfs2log    1       1       1    open/syncd    N/A
hd4                 jfs2       1       1       1    open/syncd    /
hd2                 jfs2       7       7       1    open/syncd    /usr
hd9var              jfs2       2       2       1    open/syncd    /var
hd3                 jfs2       7       7       1    open/syncd    /tmp
hd1                 jfs2       20      20      1    open/syncd    /home
hd10opt             jfs2       3       3       1    open/syncd    /opt
hd11admin           jfs2       1       1       1    open/syncd    /admin
livedump            jfs2       1       1       1    open/syncd    /var/adm/ras/livedump
lg_dumplv           sysdump    2       2       1    open/syncd    N/A
fwdump              jfs2       1       1       1    open/syncd    /var/adm/ras/platform
test1               jfs        130     130     1    open/syncd    N/A
bkclodb             jfs        50      50      1    open/syncd    N/A
rootvg_vio_1        jfs        30      30      1    closed/syncd  N/A
bkcloapp            jfs        28      28      1    open/syncd    N/A
diskpool_1:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
cloappa             jfs        280     280     1    open/syncd    N/A
ebs_backup1         jfs        116     116     1    open/syncd    N/A
paging_1            jfs        28      28      1    open/syncd    N/A
paging_2            jfs        32      32      1    open/syncd    N/A
VMLibrary           jfs2       60      60      1    open/syncd    /var/vio/VMLibrary
rootvg_6            jfs        60      60      1    open/syncd    N/A
rootvg_7            jfs        60      60      1    open/syncd    N/A
archive_log_2       jfs        40      40      1    open/syncd    N/A
diskpool_2:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
rootvg_1            jfs        60      60      1    open/syncd    N/A
rootvg_2            jfs        40      40      1    open/syncd    N/A
test_ORADB          jfs        680     680     1    open/syncd    N/A
rootvg_0            jfs        60      60      1    open/syncd    N/A
rootvg_8            jfs        60      60      1    open/syncd    N/A
test_compress       jfs        80      80      1    open/syncd    N/A
rootvg_61_3         jfs        76      76      1    open/syncd    N/A
diskpool_3:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
rootvg_53_upg       jfs        60      60      1    open/syncd    N/A
ebs_backup2         jfs        88      88      1    open/syncd    N/A
rootvg_61_2         jfs        72      72      1    open/syncd    N/A
rootvg_3            jfs        60      60      1    open/syncd    N/A
ebs_backup3         jfs        104     104     1    open/syncd    N/A
ebs_backup0         jfs        93      93      1    open/syncd    N/A
diskpool_4:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
clodba              jfs        680     680     1    open/syncd    N/A
oracle_ebs_2        jfs        628     628     2    open/syncd    N/A
disk_pool_5:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
dbrman              jfs        620     620     1    open/syncd    N/A
ORA_APP             jfs        280     280     1    open/syncd    N/A

0516-062 : Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics
#


as you can notice it is giving error because unable to get from patchtest vg.

Bingo,
The culprit is

/dev/fslv00  --         /space                jfs2  -- rw         no  no

Umount the /space filesystem
umount -f  /space

This should automatically varyoff patchtest VG.

but unable to unmount it

# umount -f  /space
umount: 0506-347 Cannot find anything to unmount.

# unmount /space
unmount: 0506-347 Cannot find anything to unmount.
#

# cat /etc/filesystems | grep space

/space:
        dev             = /dev/fslv00
        vfs             = jfs2
        log             = /dev/loglv00
        mount           = false
        options         = rw
        account         = false

Ok, remove it

It won't let me remove it
please see the output

# unmount /space
unmount: 0506-347 Cannot find anything to unmount.
# unmount /dev/fslv00
unmount: 0506-347 Cannot find anything to unmount.
# rmlv /dev/fslv00
Warning, all data contained on logical volume /dev/fslv00 will be destroyed.
rmlv: Do you wish to continue? y(es) n(o)? yes
0516-306 getlvodm: Unable to find  /dev/fslv00 in the Device
        Configuration Database.
0516-912 rmlv: Unable to remove logical volume /dev/fslv00.

# rmlv -f /dev/fslv00
0516-306 getlvodm: Unable to find  /dev/fslv00 in the Device
        Configuration Database.
0516-912 rmlv: Unable to remove logical volume /dev/fslv00.

How about if I remove it from the file /etc/filesystems the entry for /space ? and then try it or do it need to reboot the machine ?

Send me the output of

sysdumpdev -l
lsps -a
Also, try
rmfs  /space

By the way when you run rmlv, you just mention the lvname and NO /dev
So, it should be
rmlv -f fslv00

here it the output

# sysdumpdev -l
primary              /dev/lg_dumplv
secondary            /dev/sysdumpnull
copy directory       /var/adm/ras
forced copy flag     TRUE
always allow dump    FALSE
dump compression     ON
type of dump         traditional
# lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
paging00        hdisk0            rootvg        1024MB     1   yes   yes    lv     0
hd6             hdisk0            rootvg         512MB     1   yes   yes    lv     0
# rmfs /space
# rmlv -f fslv00
0516-306 getlvodm: Unable to find  fslv00 in the Device
        Configuration Database.
0516-912 rmlv: Unable to remove logical volume fslv00.
#
# lsfs
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/hd4        --         /                      jfs2  1048576 --         yes  no
/dev/hd1        --         /home                  jfs2  20971520 --         yes  no
/dev/hd2        --         /usr                   jfs2  7340032 --         yes  no
/dev/hd9var     --         /var                   jfs2  2097152 --         yes  no
/dev/hd3        --         /tmp                   jfs2  7340032 --         yes  no
/dev/hd11admin  --         /admin                 jfs2  1048576 --         yes  no
/proc           --         /proc                  procfs --      --         yes  no
/dev/hd10opt    --         /opt                   jfs2  3145728 --         yes  no
/dev/livedump   --         /var/adm/ras/livedump  jfs2  1048576 --         yes  no
/dev/fwdump     --         /var/adm/ras/platform  jfs2  1048576 --         no   no
/dev/VMLibrary  --         /var/vio/VMLibrary     jfs2  31457280 rw         yes  no


/space has been removed.

Now the next step is to remove the vg

# rmlv -f patchtest
0516-306 getlvodm: Unable to find  patchtest in the Device
        Configuration Database.
0516-912 rmlv: Unable to remove logical volume patchtest.

# exit
$ rmlv -f patchtest
Unable to find patchtest in the Device Configuration Database.

rmlv: Unable to remove logical volume patchtest.



$ oem_setup_env
l# spv
hdisk0          00c7780e79838606                    rootvg          active
hdisk1          00c7780e2e21ec86                    diskpool_4      active
hdisk2          00c7780ea5bd16bb                    diskpool_4      active
hdisk3          00c7780ee224f286                    disk_pool_5     active
hdisk4          00c7780e1b75933b                    diskpool_3      active
hdisk5          00c7780ece91bde2                    diskpool_2      active
hdisk6          00c7780ec2b65f4d                    diskpool_1      active
hdisk7          00c7780e5293914b                    None
hdisk9          00c7780e8945b5bb                    patchtest       active


# reducevg -df patchtest hdisk9 
0516-062 lqueryvg: Unable to read or write logical volume manager
        record. PV may be permanently corrupted. Run diagnostics
0516-882 reducevg: Unable to reduce volume group.

Ok,
Have you tried varyoffvg patchtest ?

If that doesn't varyoff, then you have to look into each vhost and their backing device and compare it with lsvg -l `lsvg` output.

If you see some devices from patchtest VG remove them.

1 Like