Does HACMP have bugs increasing filesystem or Logical volumes

Hello,

Does HACMP have bugs ? I have version 5.4 on AIX 6.1 and when I try to increase filesystem space or logical volume partitions which are under HACMP VG it gives me error:

# lsvg
rootvg
pr0oravg
px0oravg
pb0oravg
pr0sapvg
px0sapvg
pb0sapvg
pr1_pr2_vg
pr2_px1_vg
#

  
 # lsvg  pb0oravg
VOLUME GROUP:       pb0oravg                 VG IDENTIFIER:  000b43c20000d9000000011eac17c71a
VG STATE:           active                   PP SIZE:        512 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      398 (203776 megabytes)
MAX LVs:            512                      FREE PPs:       71 (36352  megabytes)
LVs:                14                       USED PPs:       327 (167424 megabytes)
OPEN LVs:           14                       QUORUM:         2 (Enabled)
TOTAL PVs:          2                        VG DESCRIPTORS: 3
STALE PVs:           0                        STALE PPs:      0
ACTIVE PVs:         2                        AUTO ON:        no
MAX PPs per VG:     130048
MAX PPs per PV:     1016                     MAX PVs:        128
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:           no                       BB POLICY:      relocatable
#

  
 # lsvg -l pb0oravg
pb0oravg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
orasidPB0lv         jfs2       12      12      1    open/syncd    /oracle/PB0
ora102PB0lv         jfs2       16      16      1    open/syncd    /oracle/PB0/102_64
origlogAPB0lv       jfs2       3       3       1    open/syncd     /oracle/PB0/origlogA
origlogBPB0lv       jfs2       3       3       1    open/syncd    /oracle/PB0/origlogB
mirrlogAPB0lv       jfs2       3       3       2    open/syncd    /oracle/PB0/mirrlogA
mirrlogBPB0lv       jfs2       3       3       2    open/syncd    /oracle/PB0/mirrlogB
oraarchPB0lv        jfs2       40      40      1    open/syncd     /oracle/PB0/oraarch
orareorgPB0lv       jfs2       14      14      1    open/syncd    /oracle/PB0/sapreorg
orasaparchPB0lv     jfs2       4       4       1    open/syncd    /oracle/PB0/saparch
oratracePB0lv       jfs2       4       4       1    open/syncd    /oracle/PB0/saptrace
orabackupPB0lv      jfs2       20      20      1    open/syncd     /oracle/PB0/sapbackup
oracheckPB0lv       jfs2       4       4       1    open/syncd    /oracle/PB0/sapcheck
sapdata1PB0lv       jfs2       200     200     2    open/syncd    /oracle/PB0/sapdata1
pr2_loglv04         jfs2log    1       1       1    open/syncd    N/A

  
 Command: OK            stdout: yes           stderr: no
 Before command completion, additional instructions may appear below.
 pr2serv: LOGICAL VOLUME:     sapdata1PB0lv          VOLUME GROUP:   pb0oravg
pr2serv: LV IDENTIFIER:      000b43c20000d9000000011eac17c71a.13 PERMISSION:     read/write
pr2serv: VG STATE:           active/complete        LV STATE:       opened/syncd
pr2serv: TYPE:               jfs2                   WRITE VERIFY:   off
pr2serv: MAX LPs:            512                    PP  SIZE:        512 megabyte(s)
pr2serv: COPIES:             1                      SCHED POLICY:   parallel
pr2serv: LPs:                200                    PPs:            200
pr2serv: STALE PPs:          0                      BB POLICY:      relocatable
pr2serv: INTER-POLICY:        minimum                RELOCATABLE:    yes
pr2serv: INTRA-POLICY:       middle                 UPPER BOUND:    128
pr2serv: MOUNT POINT:        /oracle/PB0/sapdata1   LABEL:          /oracle/PB0/sapdata1
pr2serv: MIRROR WRITE CONSISTENCY: on/ACTIVE
pr2serv: EACH LP COPY ON A SEPARATE PV ?: yes
pr2serv: Serialize IO ?:     NO
  
  
  
 # df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           6.00      5.64    6%    10541     1% /
/dev/hd2           5.00      2.71   46%    41735     7% /usr
/dev/hd9var        1.00      0.56   44%     1220     1% /var
/dev/hd3           5.25      5.02    5%     2746     1% /tmp
/dev/fwdump        1.00       1.00    1%        7     1% /var/adm/ras/platform
/dev/hd1           5.00      2.27   55%     2616     1% /home
/dev/hd11admin      0.25      0.25    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt       0.25      0.16   35%     1811     5%  /opt
/dev/oraclePR0lv      8.00      6.73   16%      834     1% /oracle
/dev/orasidPB0lv      6.00      6.00    1%       58     1% /oracle/PB0
/dev/ora102PB0lv      8.00      3.20   60%    19420     3% /oracle/PB0/102_64
/dev/mirrlogAPB0lv      1.50      0.72   53%        7     1% /oracle/PB0/mirrlogA
/dev/mirrlogBPB0lv      1.50      0.72   53%        8     1%  /oracle/PB0/mirrlogB
/dev/oraarchPB0lv     20.00     15.71   22%       11     1% /oracle/PB0/oraarch
/dev/origlogAPB0lv      1.50      0.71   53%       10     1% /oracle/PB0/origlogA
/dev/origlogBPB0lv      1.50      0.71   53%       10     1% /oracle/PB0/origlogB
/dev/orasaparchPB0lv      2.00      1.99    1%      281     1% /oracle/PB0/saparch
/dev/orabackupPB0lv     10.00      9.99    1%       55     1%  /oracle/PB0/sapbackup
/dev/oracheckPB0lv      2.00      1.99    1%      175     1% /oracle/PB0/sapcheck
/dev/sapdata1PB0lv    100.00      5.53   95%       80     1% /oracle/PB0/sapdata1
#

  
 Command: failed        stdout: yes           stderr: no
 Before command completion, additional instructions may appear below.
 cl_extendlv: VG pb0oravg is concurrent
  

But the VG is not in Concurrent Mode.

Is there any method to increase the size of the filesystem or logical volumes ?

It might come as a surprise but the answer is: YES! However, occasionally the bug exists between keyboard and chair. :wink:

Firstly, don't just post the error but the steps that lead to it. Secondly depending on what you want to do it might also be necessary to post information from more than one node. Using disks in a cluster usually require two or more nodes being involved.

Basically just use the appropriate cluster procedures. Remember that handling of LVM and the prerequisites are different than on a standalone server.

Yes, I did use the cluster procedures.. I tried to increase the filesystem space from smitty hacmp --> system management --> logical volume management

I'm trying to increase the filesystem or logical volume partitions on the node which is Home for it.

Well, ... the interesting part of what you did starts exactly from that cluster LVM menu onwards. From what you post I can tell so far yes, that guy entered CSPOC correctly. But the very next turn you took could have been the wrong one.

To cut a long story short could you cut and paste CSPOC'S commandlines regarding extendlv and/or chfs from smit.log? Can you run a verification and synchronisation of your cluster successfully?

I am uploading the file which has the CSPOC commands and screen shots.

As you can see the it says the VOLUME GROUP is in concurrent mode and when I go and list a concurrent VGs there is nothing
cl_extendlv: VG pb0oravg is concurrent

anyone faced similar problem ?

thanks

I think the error message is misleading. Your Big VG is not concurrent. However, the cluster seems to have a problem in recognising/handling it correctly. From HACMP 5.1 on the default type of VG for shared VG is Enhanced Concurrent Mode so I'd suggest you start with converting your old-style VG to such an ECM VG.
Stop HACMP cluster.
Varyon the VG manually on node1
# chvg -C <yourvg>
Varyoff the VG manually on node1
Varyon the VG manually on node2
# chvg -C <yourvg>
Verify and synchronise the cluster with the autorepair function enabled!
# smitty hacmp
-> Extended Configuration
-> Extended Verification and Synchronization
After a successful synchronisation start over.

Thanks, it works. the other way is that you can increase the filesystem -- normal way -- and then synchronize it, but this requires HACMP to be down.

Using Enhanced-Concurrent VGs is great, but it's recommended that the disks have their reserve policy set to "no_reserve" if possible. This will prevent any ill-behaved programs from opening a disk with reserve, thus clobbering the VG access on the other nodes.

Also, why do you believe that synchronizing the VGs requires HACMP to be down? You should be able to run "smit HACMP --> CSPOC --> Logical Volume Management --> Synchronize Shared VG Definition" at any time.

gus2000 i think that's right way. It should be done without shutting down HACMP.

clvgdats

Yes, IBM POWER HA or HACMP 5.4 / 5.5 does have bugs regarding increasing filesystems. see here

IBM IY90392: CHANGING FILESYSTEM THROUGH CSPOC FAILS AND RESULTS IN RG_MOVE - United States

APAR status

  • Closed as program error.

Error description

  • The customer changed a filesystem size through CSPOC and
    that failed, but it also resulted in moving the resource
    group to the other node.

Local fix

Problem summary

  • ****************************************************************
  • USERS AFFECTED:
  • Users of HACMP for AIX R540 performing C-SPOC operations with
  • the cluster.es.cspoc.cmds and cluster.es.cspoc.rte filesets
  • below the level of 5.4.0.2.
    ****************************************************************
  • PROBLEM DESCRIPTION:
  • C-SPOC operations on shared volume groups can result in the
  • volume group becoming inaccessible.
    ****************************************************************
  • RECOMMENDATION:
  • Install APAR IY90392.
    ****************************************************************

Problem conclusion

  • Correct code added in 551404 to update timestamp to do so
    without setting or leaving a reserve on the volume group

Temporary fix

*********

  • HIPER *

*********

filosophizer, this was exactly, what shockneck said above:

However, he continued:

In your case your VG was not in EC-Mode and with shocknecks help you got that corrected. Your posted APAR was already fixed and if you either have AIX 6.1-04 or 5.3-11 this fix would be already on your system, according to IBMs FixDist Central.

That begs the question: what are you trying to tell us?

bakunin

Thanks Bakunin and Shockneck for your help and support. Yes, I got it fixed, but you know earlier versions of HACMP like 5.1 didn't have any problem with increasing the filesystem of a non-EC VG. VG doesn't have to be in EC-Mode for increasing the filesystem through HACMP. With the fix you can increase it without the VG being in EC-Mode. It seems that this bug is in the HACMP 5.4 / 5.5