Filesystem reduction size issue

borek · May 8, 2014, 9:51am

Hi,

Need help with following case. I am trying to shring filesystem size and reduce few luns from volume group. Situation looks like that :

# 
lsvg -l data1vg
data1vg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
data1            jfs2       3200    3200    4    open/syncd    /oracle/data1
data2            jfs2       3880    3880    5    open/syncd    /oracle/data2
loglv04             jfs2log    1       1       1    open/syncd    N/A

# lsvg -p data1vg
data1vg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdiskpower10      active            809         0           00..00..00..00..00
hdiskpower7       active            809         0           00..00..00..00..00
hdiskpower8       active            2023        662         44..405..175..00..38
hdiskpower9       active            1213        1157        243..187..242..242..243
hdiskpower0       active            4046        0           00..00..00..00..00

# lsvg -m rbddata2
rbddata2:/oracle/data2
LP    PP1  PV1               PP2  PV2               PP3  PV3
0001  1994 hdiskpower0       
[...]
1993  0470 hdiskpower10      
[...]   
2425  0395 hdiskpower7       
[...]     
2825  0986 hdiskpower8       
[...]
3825  0379 hdiskpower9       
[...]	  
3880  0434 hdiskpower9      


root@uk1gcnsd06301:# lslv data2
LOGICAL VOLUME:     data2               VOLUME GROUP:   data1vg
LV IDENTIFIER:      00c629e500004c000000012e52458c71.2 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            10240                  PP SIZE:        128 megabyte(s)
COPIES:             1                      SCHED POLICY:   parallel
LPs:                3880                   PPs:            3880
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       edge                   UPPER BOUND:    128
MOUNT POINT:        /oracle/data2   LABEL:          /oracle/data2
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO
INFINITE RETRY:     no

# lsvg data1vg
VOLUME GROUP:       data1vg               VG IDENTIFIER:  00c629e500004c000000012e52458c71
VG STATE:           active                   PP SIZE:        128 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      8900 (1139200 megabytes)
MAX LVs:            512                      FREE PPs:       1819 (232832 megabytes)
LVs:                3                        USED PPs:       7081 (906368 megabytes)
OPEN LVs:           3                        QUORUM:         3 (Enabled)
TOTAL PVs:          5                        VG DESCRIPTORS: 5
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         5                        AUTO ON:        yes
MAX PPs per VG:     130048
MAX PPs per PV:     8128                     MAX PVs:        16
LTG size (Dynamic): 1024 kilobyte(s)         AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none                     INFINITE RETRY: no

While executing command :

chfs -a size=-XGB

there is very high IO acitivity on affected disks :

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0        550.0               14.4  57.0   21.0      7.5   0.4   85.2

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdiskpower0    100.0     32160.0     259.0          8     32152
hdiskpower10     0.0       0.0       0.0          0         0
hdiskpower7      0.0       0.0       0.0          0         0
hdiskpower8      0.0       0.0       0.0          0         0
hdiskpower9     49.0     32000.0     250.0      32000         0

Filesystem above hung - application is not able to write there and hung until chfs is completed (-10G take about 6 mins...so 30MB/s - same value as iostat shows)

Question is what is happening below chfs command? LPs are fragmented across all disks...but LVM should remove last LPs (from hdiskpower9) and shrink filesystem. What data is written to hdiskpower0? Where to look?

Thoughts?

rbatte1 · May 8, 2014, 10:16am

So if you've got data spread around those PPs, would you just like to lose it? The OS is moving the data to PPs that will survive the reduction. It will be doing this for every copy of an affected LP, so with mirrors, you've just doubled/tripled up.

Yes, the IO for the filesystem will lock. If you were working to keep space clear on certain PPs, how would you like it if another process looked and said "Oh lovely, a nice contiguous area of free disk to write in."

From your iostat you need to read the display clearly. Disk hdiskpower9 is reading. Writes are to hdiskpower0.

I hope that this helps your understanding, but organise downtime to take such actions as they are more complex then growing a filesystem.

Robin

borek · May 8, 2014, 10:36am

Ohh...ok. I always thought that LVM is smart enough and can do space reduction in fly (also if reduced LPs contain data).

Both filesystems are busy in < 50% - but looks like data are fragmented across all disks and LVM "lock" data during shrink process.

Is there an option to check if given LP is free or contain filesystem data?

Is there an option to do it online? Play with defrag command?

ibmtech · May 8, 2014, 10:39am

Its just more to this than file system size reduction.

You have around 1.1TB of storage, you have emc MPIO. Now,

How are the disks configured? NPIV or vscsi or direct fibre (Have you configured the fscsiX setting?)
Disk allocation is absolutely wrong, 500 GB disk will definitely degrade the performance.
Did you configure the disks queue depth values?
What version of OS are you using?
What version of oracle are you using?
If the version of Oracle and OS require you to enable CIO, was it done.

Frankly %tm_act is way beyond, generally it should be within 30

You want to reduce the size of disks and evenly strip the LV across them.

These are few of the many question to ask.

Also, can you paste the %utilization of each File system (using df -g ),

Is this a production system?

There is other way, by bringing down (and if required umount and remount the FS) the DB and try to reduce the FS (which I don't do/recommend in realtime)

borek · May 8, 2014, 1:48pm

df output looks like that - i want reduce size to +/-300G per filesystem

# df -g | grep data
/dev/data1    400.00    191.44   53%       43     1% /oracle/data1
/dev/data2    485.00    260.82   47%       29     1% /oracle/data2

iostat output was captured while chfs command was running - during normal operation is not above 30% for sure.

Regarding your questions :

NPIV
2-3. System doesn't have performance issue. LUN size depends on storage system and application above. Queue depth is optimal for powerpath devices.
What version of OS are you using? AIX 6.1TL8SP1
What version of oracle are you using? 11G - no cio option need

Can filesystem defrag be helpful here? Is there a way to match inode/filesystem block to particular LPs? Any other "online" solution?

What about assign one big lun, move all PPs by migratepv and shrink filesystem size (once lv will be on one PV).

rbatte1 · May 9, 2014, 4:56am

Using migratepv will just move the PPs. Their contents will remain the same through the process, although you are right, you can do this with the filesystem in use. The problem is, you will still have your data spread over all the PPs.

What is the eventual aim of all this anyway? It could be to free a LUN or to create another LV for another purpose. It would be good to know.

Robin

P.S. if you considered adding a mirror, synchronising them and then dropping the original, you are still going to have data across your PPs as the mirror is at the PP level.