High Disk0 & Disk1 Utilization

ggovotsis · March 12, 2008, 7:18am

Dear all,

A p505 Server with AIX 5.3 is running our backofice System.

It contains several Data Bases that its DB is on individual VG

I have a big utilization at the OS Disks,

Please see bellow topas results.

Any ideas?

Thanks in advance

George Govotsis

                                            Syscall    7307  Writech5511.5K7

Kernel 4.7 |## User 2.8 |# Wait 49.4 |############## Idle 43.1 |############# Runqueue 0.0 Dirblk 0
Network KBPS en0 2.6 en1 1.0 lo0 0.0 Steals 1997 % Comp 83.8
Disk Busy% hdisk0 100.0 2362.0 413.0 hdisk1 100.0 2372.0 415.5 hdisk2 15.0 6596.2 114.5 hdisk3 15.0 6604.2 116.5 hdisk10 9.5 hdisk23 4.0 hdisk6 4.0 hdisk7 3.5 hdisk8 0.5 hdisk9 0.5 hdisk4 0.0 hdisk11 0.0 cd0 0.0 dac2 0.0 7000.2 161.5 hdisk12 0.0 hdisk13 0.0 hdisk14 0.0 | Reads 740 Rawin 0
| Writes 3618 Ttyout 621
| Forks 0 Igets 0
| Execs 0 Namei 69
I-Pack O-Pack KB-In KB-Out Waitqueue 9.5
5.0 4.0 0.6 2.0
2.0 0.5 0.3 0.7 PAGING MEMORY
0.0 0.0 0.0 0.0 Faults 530 Real,MB 3840
KBPS TPS KB-Read KB-Writ PgspIn 60 % Noncomp 6.0
76.0 2286.0 PgspOut 876 % Client 6.0
90.0 2282.0 PageIn 160
0.0 6596.2 PageOut 2566 PAGING SPACE
8.0 6596.2 Sios 2610 Size,MB 25088
932.0 168.0 76.0 856.0 % Used 24.1
352.0 31.0 12.0 340.0 NFS (calls/sec) % Free 75.8
41.8 13.0 0.0 41.8 ServerV2 0
41.8 13.0 0.0 41.8 ClientV2 0 Press:
2.2 1.0 0.0 2.2 ServerV3 0 "h" for help
2.2 1.0 0.0 2.2 ClientV3 0 "q" to quit
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
20.0 6980.2
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0

Name PID CPU% PgSp Owner
_progres 1093776 6.0 10.4 ace
lrud 16392 0.9 0.5 root
topas 3801098 0.5 3.2 ace
_progres 2838730 0.2 10.9 ace
_progres 2289822 0.1 12.2 ace
j2pg 98360 0.1 2.0 root
java 1011810 0.1 50.8 root
java 1241188 0.0 17.1 root
pilegc 45078 0.0 7.8 root
_progres 2818288 0.0 10.2 ace
_progres 1921266 0.0 18.5 ace
_progres 3547302 0.0 18.5 ace
_progres 1962182 0.0 17.5 ace
_progres 2216082 0.0 17.5 ace
_progres 2826284 0.0 17.7 ace
_progres 1425546 0.0 11.3 ace
gil 69666 0.0 0.9 root
_progres 2891846 0.0 11.0 ace

bakunin · March 12, 2008, 10:42am

The high "wait"-value suggests that heavy swapping is going on. Run a "vmstat 1" for some time (1-2 screen full) and observe the middle columns titled "page" These should be mostly "0" with occasional little numbers mixed in. If some columns are more or less constantly non-0, then you perhaps have not enough memory.

If you have not enough memory the machine starts paging and usually the internal disks are housing the swap space. This would explain the high disk activity.

Issue a "svmon -G" (as root user) and observe the output. Here is a sample with the important parts marked bold:

# svmon -G
               size      inuse       free        pin    virtual
memory      2097152    2028714      68438     432098    1947206
pg space    1572864     225772

               work       pers       clnt
pin          432098          0          0
in use      1796594      62110     170010

PageSize   PoolSize      inuse       pgsp        pin    virtual
s   4 KB          -    1921210     217388     348450    1836022
m  64 KB          -       6719        524       5228       6949

The left number is the amount of memory you would need, the right value is the amount of memory you have. Both numbers are in 4k-units (memory pages). If the right value is way below the left one you can roughly estimate the amount of memory you will need to install by building the difference: eg. if the "inuse" shows 1Mio more than "virtual" then you will need approximately 1Mio x 4k = 4GB additional memory.

Note, that this is only a rough estimation - it doesn't replace a thorough analysis of the situation.

I hope this helps.

bakunin

ggovotsis · March 12, 2008, 10:52am

Thanks for your help I will try them and I will let you know

Thanks,

George Govotsis

shockneck · March 12, 2008, 10:56am

bakunin:

[...]

# svmon -G
   size      inuse       free        pin    virtual
memory      2097152    2028714      68438     432098    1947206
pg space    1572864     225772

   work       pers       clnt
pin          432098          0          0
in use      1796594      62110     170010
[...]

The left number is the amount of memory you would need, the right value is the amount of memory you have. [...]

Just to clarify: "size" is the amount of RAM that is installed in the system and "virtual" is the active virtual memory that is allocated/needed.

To the OP: with 4Gig of RAM and 25% usage of a 25Gig paging space you do not really wonder why your rootvg disks (where hd6 is) are busy, do you? That is a gap that probably cannot be closed by tuning vmm. Get more RAM.

bakunin · March 12, 2008, 11:50am

If i do understand you correctly this is a misconception: If a process is used very rarely (like, say, once a day) and otherwise is idling it will get swapped out. This would add to the usage numbers of the swap space but would pose no problem to the machine at all. The swapping in would take some time, true, but only once a day. After that the process would get swapped out again, until it "rests" in the swap.

Obviously this is not the case here, so my point is not connected to the problem the OP raised and of mer�ly theoretical interest. I just had a case where a customers sysdmin pointed to the output of "lsps -a" and demanded more memory whereas the machine was already having enough memory at all. This was easily shown by analysing the vmstat-output.

bakunin

shockneck · March 12, 2008, 12:09pm

I see what you mean. However, I would think that such a theoretical point might confuse the OP as what you describe does probably not apply to his p5