Bypass stale PE ?

port43 · June 9, 2014, 12:51pm

Hello,
I have an ancient HP-UX 11.11 system where I have a logical volume marked stale and I can't get it sync'd. I have tried lvsync and lvreduce/lvextend to no avail. It is just one 4Mb PE on the disk that I can't get current.

# lvdisplay -v /dev/vg00/lvol5 | grep stale
LV Status                   available/stale
   00652 /dev/dsk/c9t3d0    00069 current  /dev/dsk/c9t5d0    01470 stale

# vgdisplay -v vg00
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write
VG Status                   available
Max LV                      255
Cur LV                      9
Open LV                     9
Max PV                      16
Cur PV                      6
Act PV                      6
Max PE per PV               2500
VGDA                        12
PE Size (Mbytes)            4
Total PE                    10725
Alloc PE                    10104
Free PE                     621
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0

   --- Logical volumes ---
   ...
   LV Name                     /dev/vg00/lvol5
   LV Status                   available/stale
   LV Size (Mbytes)            3584
   Current LE                  896
   Allocated PE                1792
   Used PV                     3
   ...
   --- Physical volumes ---
   ...
   PV Name                     /dev/dsk/c9t5d0
   PV Status                   available
   Total PE                    2170
   Free PE                     56
   Autoswitch                  On

# diskinfo /dev/rdsk/c9t5d0
SCSI describe of /dev/rdsk/c9t5d0:
             vendor: SEAGATE
         product id: ST39175LC
               type: direct access
               size: 8891556 Kbytes
   bytes per sector: 512

I'm wondering if there isn't a way to mark that single PE bad so that LVM won't use it and reassign that LE to another PE since there are more available? I just find it hard to swallow that I have to replace the disk because of a 4Mb sliver.

Thank you for your attention.

Corona688 · June 9, 2014, 12:58pm

If you are having physical disk problems, it's not just the 4MB sliver that's failing. Any disk made in the last 20 years or so auto-remaps failing areas, to no longer be able to do so means so many sectors have failed all over the disk that it can't remap anymore.

port43 · June 10, 2014, 10:23am

@#%$ !! Thanks Corona688.

---------- Post updated 06-10-14 at 10:23 AM ---------- Previous update was 06-09-14 at 01:04 PM ----------

After reducing the logical volume and reducing the volume group, I replaced the disk with another one:

# diskinfo /dev/rdsk/c9t5d0
SCSI describe of /dev/rdsk/c9t5d0:
             vendor: SEAGATE
         product id: ST39173WC
               type: direct access
               size: 8891556 Kbytes
   bytes per sector: 512

backed up /etc/lvmtab, ran vgscan, pvcreate, vgextend, lvextend, and I have the same exact stale PE:

manwms01:/root # lvdisplay -v /dev/vg00/lvol5 | grep stale
LV Status                   available/stale
   00652 /dev/dsk/c9t3d0    00069 current  /dev/dsk/c9t5d0    01470 stale

How is that even possible?

vbe · June 10, 2014, 10:55am

We know so little about your system from the sparse information you gave that it is almost impossible to answer..

port43 · June 10, 2014, 11:10am

What other information can I provide that would help? I'll pass on any information I can.

This is an old N4000 and the disks are in a Jamaica enclosure. None of it is supported by HP anymore so I'm left to try and make it limp along.

vbe · June 10, 2014, 11:22am

OK
N4000 had slots fo internal disks, are you using them?
I have no more HP-UX boxes now so I have try to do things by memory...
What I find strange is have vg00 ( OS ) with 6 disks... is it you never made other VG?
is there any mirroring?
Can sou give use the output of

 ioscan -funC disk

and look if there is anything interresting in /var/adm/syslog/syslog.log, you may be having a failing controller or a failing disk sending weird stuff...
What does pvdisplay -v /dev/dsk/c9t5d0 give you?
But the best would be you, describe us vg00, how it is configured, how the disks are set using what interface/controllers... And giving a minimum information of FS layout...

port43 · June 10, 2014, 12:03pm

I inherited the server support and the people responsible for the initial setup are long gone so I do not know much of the history and the server is located in another part of the country where I have no "physical" access to it.

There are no internal disks in the server; it's root volume group is on six of eight Seagate disks in the external A3312A (Jamaica) enclosure. There are two other volume groups but they are in an EMC disk array.

 # ioscan -funC disk
Class     I  H/W Path       Driver S/W State   H/W Type     Description
=======================================================================
disk      0  0/0/1/0.6.0    sdisk CLAIMED     DEVICE       HP      DVD-ROM 6x/32x
                           /dev/dsk/c0t6d0   /dev/rdsk/c0t6d0
disk      1  0/8/0/0.1.0    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t1d0   /dev/rdsk/c4t1d0
disk      2  0/8/0/0.1.1    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t1d1   /dev/rdsk/c4t1d1
disk      3  0/8/0/0.1.2    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t1d2   /dev/rdsk/c4t1d2
disk      4  0/8/0/0.1.3    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t1d3   /dev/rdsk/c4t1d3
disk      5  0/8/0/0.1.4    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t1d4   /dev/rdsk/c4t1d4
disk      6  0/8/0/0.1.5    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t1d5   /dev/rdsk/c4t1d5
disk      7  0/8/0/0.3.0    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t3d0   /dev/rdsk/c4t3d0
disk      8  0/8/0/0.3.1    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t3d1   /dev/rdsk/c4t3d1
disk      9  0/8/0/0.3.2    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t3d2   /dev/rdsk/c4t3d2
disk     10  0/8/0/0.3.3    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t3d3   /dev/rdsk/c4t3d3
disk     11  0/8/0/0.3.4    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t3d4   /dev/rdsk/c4t3d4
disk     12  0/8/0/0.3.5    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t3d5   /dev/rdsk/c4t3d5
disk     13  0/8/0/0.11.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t11d0   /dev/rdsk/c4t11d0
disk     14  0/8/0/0.11.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t11d1   /dev/rdsk/c4t11d1
disk     15  0/8/0/0.11.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t11d2   /dev/rdsk/c4t11d2
disk     16  0/8/0/0.11.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t11d3   /dev/rdsk/c4t11d3
disk     17  0/8/0/0.11.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t11d4   /dev/rdsk/c4t11d4
disk     18  0/8/0/0.13.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t13d0   /dev/rdsk/c4t13d0
disk     19  0/8/0/0.13.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t13d1   /dev/rdsk/c4t13d1
disk     20  0/8/0/0.13.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t13d2   /dev/rdsk/c4t13d2
disk     21  0/8/0/0.13.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t13d3   /dev/rdsk/c4t13d3
disk     22  0/8/0/0.13.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t13d4   /dev/rdsk/c4t13d4
disk     23  0/8/0/0.15.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t15d0   /dev/rdsk/c4t15d0
disk     24  0/8/0/0.15.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t15d1   /dev/rdsk/c4t15d1
disk     25  0/8/0/0.15.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t15d2   /dev/rdsk/c4t15d2
disk     26  0/8/0/0.15.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t15d3   /dev/rdsk/c4t15d3
disk     27  0/8/0/0.15.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t15d4   /dev/rdsk/c4t15d4
disk     28  0/8/0/0.15.5   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c4t15d5   /dev/rdsk/c4t15d5
disk     29  0/10/0/0.2.0   sdisk CLAIMED     DEVICE       SEAGATE ST34371N
                           /dev/dsk/c5t2d0   /dev/rdsk/c5t2d0
disk     30  0/10/0/0.3.0   sdisk CLAIMED     DEVICE       SEAGATE ST34572WC
                           /dev/dsk/c5t3d0   /dev/rdsk/c5t3d0
disk     31  0/10/0/0.4.0   sdisk CLAIMED     DEVICE       SEAGATE ST39175LC
                           /dev/dsk/c5t4d0   /dev/rdsk/c5t4d0
disk     32  0/10/0/0.5.0   sdisk CLAIMED     DEVICE       SEAGATE ST39173WC
                           /dev/dsk/c5t5d0   /dev/rdsk/c5t5d0
disk     33  0/12/0/0.0.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t0d0   /dev/rdsk/c6t0d0
disk     34  0/12/0/0.0.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t0d1   /dev/rdsk/c6t0d1
disk     35  0/12/0/0.0.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t0d2   /dev/rdsk/c6t0d2
disk     36  0/12/0/0.0.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t0d3   /dev/rdsk/c6t0d3
disk     37  0/12/0/0.0.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t0d4   /dev/rdsk/c6t0d4
disk     38  0/12/0/0.0.5   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t0d5   /dev/rdsk/c6t0d5
disk     39  0/12/0/0.2.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t2d0   /dev/rdsk/c6t2d0
disk     40  0/12/0/0.2.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t2d1   /dev/rdsk/c6t2d1
disk     41  0/12/0/0.2.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t2d2   /dev/rdsk/c6t2d2
disk     42  0/12/0/0.2.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t2d3   /dev/rdsk/c6t2d3
disk     43  0/12/0/0.2.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t2d4   /dev/rdsk/c6t2d4
disk     44  0/12/0/0.2.5   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t2d5   /dev/rdsk/c6t2d5
disk     45  0/12/0/0.10.0  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t10d0   /dev/rdsk/c6t10d0
disk     46  0/12/0/0.10.1  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t10d1   /dev/rdsk/c6t10d1
disk     47  0/12/0/0.10.2  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t10d2   /dev/rdsk/c6t10d2
disk     48  0/12/0/0.10.3  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t10d3   /dev/rdsk/c6t10d3
disk     49  0/12/0/0.10.4  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t10d4   /dev/rdsk/c6t10d4
disk     50  0/12/0/0.12.0  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t12d0   /dev/rdsk/c6t12d0
disk     51  0/12/0/0.12.1  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t12d1   /dev/rdsk/c6t12d1
disk     52  0/12/0/0.12.2  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t12d2   /dev/rdsk/c6t12d2
disk     53  0/12/0/0.12.3  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t12d3   /dev/rdsk/c6t12d3
disk     54  0/12/0/0.12.4  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t12d4   /dev/rdsk/c6t12d4
disk     55  0/12/0/0.15.0  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t15d0   /dev/rdsk/c6t15d0
disk     56  0/12/0/0.15.1  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t15d1   /dev/rdsk/c6t15d1
disk     57  0/12/0/0.15.2  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t15d2   /dev/rdsk/c6t15d2
disk     58  0/12/0/0.15.3  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t15d3   /dev/rdsk/c6t15d3
disk     59  0/12/0/0.15.4  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t15d4   /dev/rdsk/c6t15d4
disk     60  0/12/0/0.15.5  sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c6t15d5   /dev/rdsk/c6t15d5
disk     61  1/2/0/0.1.0    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t1d0   /dev/rdsk/c7t1d0
disk     62  1/2/0/0.1.1    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t1d1   /dev/rdsk/c7t1d1
disk     63  1/2/0/0.1.2    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t1d2   /dev/rdsk/c7t1d2
disk     64  1/2/0/0.1.3    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t1d3   /dev/rdsk/c7t1d3
disk     65  1/2/0/0.1.4    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t1d4   /dev/rdsk/c7t1d4
disk     66  1/2/0/0.1.5    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t1d5   /dev/rdsk/c7t1d5
disk     67  1/2/0/0.3.0    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t3d0   /dev/rdsk/c7t3d0
disk     68  1/2/0/0.3.1    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t3d1   /dev/rdsk/c7t3d1
disk     69  1/2/0/0.3.2    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t3d2   /dev/rdsk/c7t3d2
disk     70  1/2/0/0.3.3    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t3d3   /dev/rdsk/c7t3d3
disk     71  1/2/0/0.3.4    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t3d4   /dev/rdsk/c7t3d4
disk     72  1/2/0/0.3.5    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t3d5   /dev/rdsk/c7t3d5
disk     73  1/2/0/0.11.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t11d0   /dev/rdsk/c7t11d0
disk     74  1/2/0/0.11.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t11d1   /dev/rdsk/c7t11d1
disk     75  1/2/0/0.11.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t11d2   /dev/rdsk/c7t11d2
disk     76  1/2/0/0.11.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t11d3   /dev/rdsk/c7t11d3
disk     77  1/2/0/0.11.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t11d4   /dev/rdsk/c7t11d4
disk     78  1/2/0/0.13.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t13d0   /dev/rdsk/c7t13d0
disk     79  1/2/0/0.13.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t13d1   /dev/rdsk/c7t13d1
disk     80  1/2/0/0.13.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t13d2   /dev/rdsk/c7t13d2
disk     81  1/2/0/0.13.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t13d3   /dev/rdsk/c7t13d3
disk     82  1/2/0/0.13.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t13d4   /dev/rdsk/c7t13d4
disk     83  1/2/0/0.15.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t15d0   /dev/rdsk/c7t15d0
disk     84  1/2/0/0.15.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t15d1   /dev/rdsk/c7t15d1
disk     85  1/2/0/0.15.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t15d2   /dev/rdsk/c7t15d2
disk     86  1/2/0/0.15.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t15d3   /dev/rdsk/c7t15d3
disk     87  1/2/0/0.15.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t15d4   /dev/rdsk/c7t15d4
disk     88  1/2/0/0.15.5   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c7t15d5   /dev/rdsk/c7t15d5
disk     89  1/4/0/0.0.0    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t0d0   /dev/rdsk/c8t0d0
disk     90  1/4/0/0.0.1    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t0d1   /dev/rdsk/c8t0d1
disk     91  1/4/0/0.0.2    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t0d2   /dev/rdsk/c8t0d2
disk     92  1/4/0/0.0.3    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t0d3   /dev/rdsk/c8t0d3
disk     93  1/4/0/0.0.4    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t0d4   /dev/rdsk/c8t0d4
disk     94  1/4/0/0.0.5    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t0d5   /dev/rdsk/c8t0d5
disk     95  1/4/0/0.2.0    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t2d0   /dev/rdsk/c8t2d0
disk     96  1/4/0/0.2.1    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t2d1   /dev/rdsk/c8t2d1
disk     97  1/4/0/0.2.2    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t2d2   /dev/rdsk/c8t2d2
disk     98  1/4/0/0.2.3    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t2d3   /dev/rdsk/c8t2d3
disk     99  1/4/0/0.2.4    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t2d4   /dev/rdsk/c8t2d4
disk    100  1/4/0/0.2.5    sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t2d5   /dev/rdsk/c8t2d5
disk    101  1/4/0/0.10.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t10d0   /dev/rdsk/c8t10d0
disk    102  1/4/0/0.10.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t10d1   /dev/rdsk/c8t10d1
disk    103  1/4/0/0.10.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t10d2   /dev/rdsk/c8t10d2
disk    104  1/4/0/0.10.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t10d3     /dev/dsk/disk_query  /dev/rdsk/c8t10d3
disk    105  1/4/0/0.10.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t10d4   /dev/rdsk/c8t10d4
disk    106  1/4/0/0.12.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t12d0   /dev/rdsk/c8t12d0
disk    107  1/4/0/0.12.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t12d1   /dev/rdsk/c8t12d1
disk    108  1/4/0/0.12.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t12d2   /dev/rdsk/c8t12d2
disk    109  1/4/0/0.12.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t12d3   /dev/rdsk/c8t12d3
disk    110  1/4/0/0.12.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t12d4   /dev/rdsk/c8t12d4
disk    111  1/4/0/0.15.0   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t15d0   /dev/rdsk/c8t15d0
disk    112  1/4/0/0.15.1   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t15d1   /dev/rdsk/c8t15d1
disk    113  1/4/0/0.15.2   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t15d2   /dev/rdsk/c8t15d2
disk    114  1/4/0/0.15.3   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t15d3   /dev/rdsk/c8t15d3
disk    115  1/4/0/0.15.4   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t15d4   /dev/rdsk/c8t15d4
disk    116  1/4/0/0.15.5   sdisk CLAIMED     DEVICE       EMC     SYMMETRIX
                           /dev/dsk/c8t15d5   /dev/rdsk/c8t15d5
disk    117  1/10/0/0.2.0   sdisk CLAIMED     DEVICE       SEAGATE ST34573WC
                           /dev/dsk/c9t2d0   /dev/rdsk/c9t2d0
disk    118  1/10/0/0.3.0   sdisk CLAIMED     DEVICE       SEAGATE ST34572WC
                           /dev/dsk/c9t3d0   /dev/rdsk/c9t3d0
disk    119  1/10/0/0.4.0   sdisk CLAIMED     DEVICE       SEAGATE ST39175LC
                           /dev/dsk/c9t4d0   /dev/rdsk/c9t4d0
disk    120  1/10/0/0.5.0   sdisk CLAIMED     DEVICE       SEAGATE ST39173WC
                           /dev/dsk/c9t5d0   /dev/rdsk/c9t5d0

The pvdisplay output is 2198 lines but it is basically:

# pvdisplay -v /dev/dsk/c9t5d0

--- Physical volumes ---
PV Name                     /dev/dsk/c9t5d0
VG Name                     /dev/vg00
PV Status                   available
Allocatable                 yes
VGDA                        2
Cur LV                      7
PE Size (Mbytes)            4
Total PE                    2170
Free PE                     0
Allocated PE                2170
Stale PE                    1
IO Timeout (Seconds)        default
Autoswitch                  On

   --- Distribution of physical volume ---
   LV Name            LE of LV  PE for LV
   /dev/vg00/lvol1    50        50
   /dev/vg00/lvol2    256       256
   /dev/vg00/lvol3    256       256
   /dev/vg00/lvol4    256       256
   /dev/vg00/lvol5    896       896
   /dev/vg00/lvol6    256       256
   /dev/vg00/lvol8    200       200

   --- Physical extents ---
   PE    Status   LV                 LE
   00000 current  /dev/vg00/lvol1    00000
   00001 current  /dev/vg00/lvol1    00001
   00002 current  /dev/vg00/lvol1    00002
   00003 current  /dev/vg00/lvol1    00003
   ... (more of the same for lvols 2, 3, 4, and 5)
   01465 current  /dev/vg00/lvol5    00647
   01466 current  /dev/vg00/lvol5    00648
   01467 current  /dev/vg00/lvol5    00649
   01468 current  /dev/vg00/lvol5    00650
   01469 current  /dev/vg00/lvol5    00651
   01470 stale    /dev/vg00/lvol5    00652
   01471 current  /dev/vg00/lvol5    00653
   01472 current  /dev/vg00/lvol5    00654
   01473 current  /dev/vg00/lvol5    00655
   ...(more of the same for lvols 5, 6, and 8)
   02165 current  /dev/vg00/lvol8    00195
   02166 current  /dev/vg00/lvol8    00196
   02167 current  /dev/vg00/lvol8    00197
   02168 current  /dev/vg00/lvol8    00198
   02169 current  /dev/vg00/lvol8    00199

Aside from typical su, login, ftp, and telnet information, the only thing in the syslog that looks like it pertains to this is the continuous (along with the vgextend and lvextend commands I executed):

WARNING:vmunix: SCSI: lvstat on servername: Tue Jun 10 05:20:16 CDT 2014
The following logical volumes are not current:
        Logical Volume            Physical Volume     Status
        ---------------------------------------------------------
        /dev/vg00/lvol5           /dev/dsk/c9t5d0     stale

and that is the same output from the previous disk replacement as well as the current one.

vgdisplay for vg00 (where the stale PE is):

# vgdisplay -v vg00
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write
VG Status                   available
Max LV                      255
Cur LV                      9
Open LV                     9
Max PV                      16
Cur PV                      6
Act PV                      6
Max PE per PV               2500
VGDA                        12
PE Size (Mbytes)            4
Total PE                    10725
Alloc PE                    10104
Free PE                     621
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0

   --- Logical volumes ---
   LV Name                     /dev/vg00/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            200
   Current LE                  50
   Allocated PE                100
   Used PV                     2

   LV Name                     /dev/vg00/lvol2
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  256
   Allocated PE                512
   Used PV                     2

   LV Name                     /dev/vg00/lvol3
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  256
   Allocated PE                512
   Used PV                     2

   LV Name                     /dev/vg00/lvol4
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  256
   Allocated PE                512
   Used PV                     2

   LV Name                     /dev/vg00/lvol5
   LV Status                   available/stale
   LV Size (Mbytes)            3584
   Current LE                  896
   Allocated PE                1792
   Used PV                     3

   LV Name                     /dev/vg00/lvol6
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  256
   Allocated PE                512
   Used PV                     2

   LV Name                     /dev/vg00/lvol7
   LV Status                   available/syncd
   LV Size (Mbytes)            2048
   Current LE                  512
   Allocated PE                1024
   Used PV                     3

   LV Name                     /dev/vg00/lvol8
   LV Status                   available/syncd
   LV Size (Mbytes)            1600
   Current LE                  400
   Allocated PE                800
   Used PV                     4

   LV Name                     /dev/vg00/lvswap
   LV Status                   available/syncd
   LV Size (Mbytes)            8680
   Current LE                  2170
   Allocated PE                4340
   Used PV                     2


   --- Physical volumes ---
   PV Name                     /dev/dsk/c5t3d0
   PV Status                   available
   Total PE                    1023
   Free PE                     311
   Autoswitch                  On

   PV Name                     /dev/dsk/c5t4d0
   PV Status                   available
   Total PE                    2170
   Free PE                     0
   Autoswitch                  On

   PV Name                     /dev/dsk/c5t5d0
   PV Status                   available
   Total PE                    2169
   Free PE                     0
   Autoswitch                  On

   PV Name                     /dev/dsk/c9t3d0
   PV Status                   available
   Total PE                    1023
   Free PE                     310
   Autoswitch                  On

   PV Name                     /dev/dsk/c9t4d0
   PV Status                   available
   Total PE                    2170
   Free PE                     0
   Autoswitch                  On

   PV Name                     /dev/dsk/c9t5d0
   PV Status                   available
   Total PE                    2170
   Free PE                     0
   Autoswitch                  On

The file systems in vg00:

# bdf | grep vg00
/dev/vg00/lvol3    1048576  230163  767292   23% /
/dev/vg00/lvol1     199381   53441  126001   30% /stand
/dev/vg00/lvol8    1638400 1311776  312468   81% /var
/dev/vg00/lvol7    2097152 1457564  599631   71% /usr
/dev/vg00/lvol6    1048576   38410  948350    4% /tmp
/dev/vg00/lvol5    3584000 3017016  533652   85% /opt
/dev/vg00/lvol4    1048576  638589  390920   62% /home

 # df | grep vg00
/home                (/dev/vg00/lvol4     ):   782888 blocks    102636 i-nodes
/opt                 (/dev/vg00/lvol5     ):  1067304 blocks    141745 i-nodes
/tmp                 (/dev/vg00/lvol6     ):  1896686 blocks    252538 i-nodes
/usr                 (/dev/vg00/lvol7     ):  1199262 blocks    159894 i-nodes
/var                 (/dev/vg00/lvol8     ):   624936 blocks     81655 i-nodes
/stand               (/dev/vg00/lvol1     ):   252002 blocks     32190 i-nodes
/                    (/dev/vg00/lvol3     ):  1534584 blocks    204602 i-nodes

I am not sure how to get the information

vbe · June 10, 2014, 12:34pm

From what I see you are in mirror using 2 controllers : 0/10/0... and 1/10/0...
If you look, all your LV are like :

   Current LE                  256
   Allocated PE                512

PE is twice the value of LE ... except the one giving you trouble... only Im a bit stressed now at work, will try to finish rapidly and if no one else had helped you, join back from my laptop at home...

All the best

port43 · June 10, 2014, 1:05pm

Ah, yes, there are three disks in use on each controller. Whoever initially set this up wasn't thinking and didn't keep the mirrors on distinct controllers since some volumes have data on both sides but the mirror on the other. For instance:

 # lvdisplay -v /dev/vg00/lvol5
--- Logical volumes ---
LV Name                     /dev/vg00/lvol5
VG Name                     /dev/vg00
LV Permission               read/write
LV Status                   available/stale
Mirror copies               1
Consistency Recovery        MWC
Schedule                    parallel
LV Size (Mbytes)            3584
Current LE                  896
Allocated PE                1792
Stripes                     0
Stripe Size (Kbytes)        0
Bad block                   on
Allocation                  strict
IO Timeout (Seconds)        default

   --- Distribution of logical volume ---
   PV Name            LE on PV  PE on PV
   /dev/dsk/c5t5d0    583       583
   /dev/dsk/c9t3d0    313       313
   /dev/dsk/c9t5d0    896       896

   --- Logical extents ---
   LE    PV1                PE1   Status 1 PV2                PE2   Status 2
   00000 /dev/dsk/c5t5d0    00818 current  /dev/dsk/c9t5d0    00818 current
   00001 /dev/dsk/c5t5d0    00819 current  /dev/dsk/c9t5d0    00819 current
   00002 /dev/dsk/c5t5d0    00820 current  /dev/dsk/c9t5d0    00820 current
   00003 /dev/dsk/c5t5d0    00821 current  /dev/dsk/c9t5d0    00821 current
   ... (more of the same until)
   00579 /dev/dsk/c5t5d0    02165 current  /dev/dsk/c9t5d0    01397 current
   00580 /dev/dsk/c5t5d0    02166 current  /dev/dsk/c9t5d0    01398 current
   00581 /dev/dsk/c5t5d0    02167 current  /dev/dsk/c9t5d0    01399 current
   00582 /dev/dsk/c5t5d0    02168 current  /dev/dsk/c9t5d0    01400 current
   00583 /dev/dsk/c9t3d0    00000 current  /dev/dsk/c9t5d0    01401 current
   00584 /dev/dsk/c9t3d0    00001 current  /dev/dsk/c9t5d0    01402 current
   00585 /dev/dsk/c9t3d0    00002 current  /dev/dsk/c9t5d0    01403 current
   00586 /dev/dsk/c9t3d0    00003 current  /dev/dsk/c9t5d0    01404 current
   ... (and still more up to the stale PE)
   00650 /dev/dsk/c9t3d0    00067 current  /dev/dsk/c9t5d0    01468 current
   00651 /dev/dsk/c9t3d0    00068 current  /dev/dsk/c9t5d0    01469 current
   00652 /dev/dsk/c9t3d0    00069 current  /dev/dsk/c9t5d0    01470 stale
   00653 /dev/dsk/c9t3d0    00070 current  /dev/dsk/c9t5d0    01471 current
   00654 /dev/dsk/c9t3d0    00071 current  /dev/dsk/c9t5d0    01472 current
   ... (and on until the bottom)
   00893 /dev/dsk/c9t3d0    00710 current  /dev/dsk/c9t5d0    01711 current
   00894 /dev/dsk/c9t3d0    00711 current  /dev/dsk/c9t5d0    01712 current
   00895 /dev/dsk/c9t3d0    00712 current  /dev/dsk/c9t5d0    01713 current

So if I lose controller 1/10/0 I lose lvol5. But that's a whole other issue.

vbe · June 10, 2014, 3:35pm

I wonder if its not because the strict allocation, that is not truly suitable when using 3 disks� Doesn't pvdisplay show anything?

port43 · June 11, 2014, 9:44am

The pvdisplay is on the first page of this thread and looks fine to me, except for the one stale part.

I looked into the fact that it has a "strict" allocation policy and from what I glean from the lvchange command documentation, that just means that the mirror copies of extents cannot share the same physical volume, which makes sense because you wouldn't want your data mirrored on the same disk because if it failed ...

         -s strict                Set the strict allocation policy.
                                    Mirror copies of a logical extent can be
                                    allocated to share or not share the same
                                    physical volume or physical volume
                                    group.  This option only makes sense
                                    when the physical volumes of the volume
                                    group that owns the specified logical
                                    volume reside on different physical
                                    disks.  strict can have one of the
                                    following values:

                                    y    Set a strict allocation policy.
                                         Mirrors of a logical extent cannot
                                         share the same physical volume.

                                    g    Set a PVG-strict allocation policy.
                                         Mirrors of a logical extent cannot
                                         share the same physical volume
                                         group.

                                    n    Do not set a strict or a PVG-strict
                                         allocation policy.  Mirrors of a
                                         logical extent can share the same
                                         physical volume.

                                    When a logical volume is mirrored, the
                                    following changes are not allowed:

                                    +    From nonstrict to strict
                                    +    From nonstrict to PVG-strict
                                    +    From strict to PVG-strict

I thought I was onto something when I saw the -C contiguous option so I could skip the bad extent but that's not available for the root volume group.

vbe · June 11, 2014, 10:21am

I never had stale on HP-UX... (cant say that of the aix's I was in charge...) So I am trying to interpolate from my bad AIX experience... About Contiguous - You do have some... HP-UX11.11 have in vg00 3 Lvols that use contiguous: the swap, root and stand, but its not the case here, how did you change disk? (what commands were passed...) because I think I did have something of the sort with a D class when I forgot to do a ??? forgot.. trying hard to remember you can help since I have no more documentation, you should do a ioscan ( full in doubt, not funC...) to update devs but there is something else I wonder if its not insf -e can you check what that does? because my idea would be to remove the disk if you are sure it is good with no bad blocks etc.. the rm all associated files /dev/rdsk/c.... then add the disk do an insf if its the right command and see if that changes after mirroring again

port43 · June 11, 2014, 11:05am

I always use the HP doc ""When Good Disks Go Bad"
and my steps boil down to:

lvreduce
vgreduce
mv /etc/lvmtab
pull disk
rmsf -H 1/10/0/0.5.0
put in new disk
ioscan to verify disk is there
insf
vgscan
pvcreate
vgextend
lvextend

I'm going to call HP and see if I can't beg my way to some help.

vbe · June 11, 2014, 11:29am

You do have a disk that is not like the others: c5t5d0 which has 2169 instead of 2170 so there is a bad block there and I wonder if that is not what is giving trouble

port43 · June 11, 2014, 6:24pm

So, I decided to unmirror everything and reorganize the logical volumes since some of them were spread over disks on the same controller or mirrored onto the same physical volume.

I got it to where I had all logical volumes on disks on one controller ie. each individual volume was completely on one disk on the 1/10/0 controller. I even put lvol5 (that had the stale PE) on it's own disk (c9t3d0) with nothing else on it. Everything was fine, all PE's were current. I then mirrored each LV to it's partner disk on the 0/10/0 controller ie. everything on c9t5d0 mirrored to c5t5d0 with no issue (lvol1 - lvol4, lvol6 - lvol8) but when I tried to mirror lvol5 from c9t3d0 to c5t3d0 I ran into the stale partition again, with the same PE's, and it's different disks this time:

# lvextend -m 1 /dev/vg00/lvol5 /dev/dsk/c5t3d0
The newly allocated mirrors are now being synchronized. This operation will
take some time. Please wait ....

lvextend: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
lvextend: Couldn't resynchronize logical volume "/dev/vg00/lvol5".
# lvdisplay -v /dev/vg00/lvol5 | grep stale
LV Status                   available/stale
   00652 /dev/dsk/c9t3d0    00069 current  /dev/dsk/c5t3d0    00652 stale

I obviously do not understand how a stale PE on one disk can be the same stale PE on another.

Could it be as simple as a bad data in lvol5 where I need to fsck that file system before trying to mirror?

vbe · June 12, 2014, 11:03am

So it must be something wrong with /dev/dsk/c9t3d0 a corrupt block that it cannot detect and shows up only the other side when trying to sync?

port43 · June 12, 2014, 3:59pm

In an amazing display of Murphy's Law, this is no longer an issue as the site took a power hit last night and the server is fried. And no, I was nowhere near it.

vbe · June 12, 2014, 4:38pm

Ah well we tried our best...