Hi all..

I need some comments or advices about my HP-UX 11.11 performance.
It runs Oracle, some apps and weblogic.

For several times in a day, it's performance drops poorly (approx for minutes until an hour).
I tried to find articles about unix, oracle and weblogic tuning, which leads me to some informations :

1. Some Unix Kernel configuration (max_thread_proc, maxfiles, maxusers) needs to be set larger than the default configurations (I will reconfigure this soon)

2. My kernel configuration : lotsfree, desfree, minfree sets to 0 (zero)
   I found some article which told to set it, according to my memory size. Anyone tried this before? did it give you a good result?

3. I ran vmstat for about 10 minutes (@1 sec), which gave me this results :

• SR : 20% return large page scan rate (avr. 9k, max 60k) and 80% return below 200 pages (most articles say this is not good, needs more RAM)
• RE : 20% return large reclaims (arg 800, max 4k) and 80% return below 10 pages
• R : 4% return more than my CPUs count (avr 10, during large SR) and 96% return below my CPUs count
• W : 5% return avr 15 pages, and 95% return almost none
• and PI also increase while RE increase

I need some comments or advices about 3 points above,
and, is it just configuration problem (tuning problem only) or I 'extremly' need more RAM?

(I'm kinda new at this, need much advices and guidances)
Thank you for your time, comments and advices.

We need to know a bit more..
What is the output of sysdef, how much RAM and what kind of disks are attached, what is the output of swapinfo -tam,

What is the size of the SGA

>output of sysdef
NAME VALUE
acctresume 4
acctsuspend 2
allocate_fs_swapmap 0
bufpages 416197
create_fastlinks 0
dbc_max_pct 50
dbc_min_pct 5
default_disk_ir 0
dskless_node 0
eisa_io_estimate 768
eqmemsize 46
file_pad 10
fs_async 0
hpux_aes_override 0
maxdsiz 262144
maxdsiz_64bit 524288
maxfiles 60
maxfiles_lim 1024
maxssiz 32768
maxssiz_64bit 262144
maxswapchunks 16384
maxtsiz 16384
maxtsiz_64bit 262144
maxuprc 3686
maxvgs 10
msgmap 4095
nbuf 673506
ncallout 7200
ncdnode 150
ndilbuffers 30
netisr_priority -1
netmemmax 0
nfile 63890
nflocks 4096
ninode 35016
no_lvm_disks 0
nproc 4096
npty 60
nstrpty 60
nswapdev 10
nswapfs 10
public_shlibs 1
remote_nfs_swap 0
rtsched_numpri 32
sema 0
semmap 4095
shmem 0
shmmni 1024
streampipes 0
swapmem_on 1
swchunk 2048
timeslice 10
unlockable_mem 16584

>how much RAM
8 Gb

>what kind of disks are attached
SCSI with fiber cable

>what is the output of swapinfo -tam,

(during performance ok)
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI
dev 12000 1012 10988 8% 0 - 1
reserve - 5081 -5081
memory 6285 1906 4379 30%
total 18285 7999 10286 44% - 0 -
(do I need to check it again during bad performance?)

>What is the size of the SGA
Fixed Size 740,232
Variable Size 738,197,504
Database Buffers 1,610,612,736
Redo Buffers 1,323,008
Total (SGA) 2,350,873,480

Thanks.

also, I notice that when the performance starts to slowdown,
the FREE in vmstat also drop below 10Mb (below 10,000 in vmstat display), which in other conditions, the value various between 40-230Mb.

the FREE also increase while SR, RE and IN increase.
any comments or advices? thanks.

before I give you my 2 cents, what kind of disk subsystem, dual path? size of volume group (approx... (idea of db size..) how many processors (2-4?) and have you some spare space on subsystem...

Forgot: in /etc/fstab how are mounted your FS (especially data) are they all vxfs?

Have you notices the slowdown when running batches (vs oltp... e.g. is it when big jobs are running, compared to many users doing queries...)

all FS mounted with vxfs

I'm not quite sure with the meaning of disk subsystem or volume group, but perhaps, this information can give you a clue (bdf command) :
Mount kbytes perc-used
/ 1,048Mb 11%
/tmp 4,096Mb 19%
/home 24,576Mb 86%
/data 425,984Mb 46%

total DB size around 100Gb (currently allocated 110Gb)

3 procs

yes, the last thing I need to check is the users session resource usage (no idea yet), but as far as I know, there is no background job running on the server during daytime.

also, some site recommended to see the x$ksmsp (related to SGA), but they dont
explain how to read them, can you give me some information? (I want to know is my current SGA size is oversize than what we need)

select sum(ksmchsiz) Bytes, ksmchcls Status
from x$ksmsp
group by ksmchcls;

R-free 132,864,168
R-freea 138,840
free 2,079,208
freeabl 308,242,376
perm 74,995,696
recr 85,657,184

Thank you very much.

the reason I started to look at the SGA, is because from the 'top command', I see some of processes are used by Oracle, which each one take 2,391Mb.
although most of it are sleeping (%CPU and %WCPU below 1%), I'm guessing that take lots of server resources too.

About /etc/fstab, there are mount option which say how the FS are tuned (tmplog,nolog delaylog etc...)
You have here some optimization that could be done but for that we would need to know the type/caracteristics of the disk-subsystem (like hardware RAID5 /cache centric...) or you may generate other issues...
So until you find out this we will put aside...

SWAP
A system will see his performance drop if it needs to swap (your case...) and is using disk swap space..
creating more swap of same size on other devices will balance I/O load...

The system buffers its read/write, oracle also, and worst maybe your subsystem also...
That means that before a modification, or read is done oracle will look in its buffer, then the system, at at last the sub-system, now I have noticed that better can be enemy of good, e.g. to much buffer in oracle will slow the system because of time parsing, compared to go and get the data...
If you know your subsystem has cache then reduce the amount of system buffers

Kernel params:
bufpages 0
dbc_max_pct 8
dbc_min_pct 5
swapmem_on 0

I cant find your values of maxusers... how many concurrent users?
I there a good reason for maxuprc 3686? (like: We all connect using oracle account...) It a heck alot for one user and can cost you as much if you fall on bad written code (infinite loops etc...) on big systems here with OA running (and because of java ) we set it at 1000 then I monitor to see if more is really needed and in most cases, not...

I would reduce of 200000 your redo and data buffers

Can you give this a try and lets see what happens?
(and look what you have in /etc/fstab, and what disks you are attached to...)

Good luck

> the type/caracteristics of the disk-subsystem (like hardware RAID5 /cache centric...)
I can't find any information about that, about /etc/fstab :
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol4 /tmp vxfs delaylog 0 2
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2
/dev/vg01/lvol1 /<mydata> vxfs rw,suid,largefiles,delaylog,datainlog 0 2

> creating more swap of same size on other devices will balance I/O load...
Ok. I will try to tune the Unix Kernel configuration on the first phase,
on the second phase, I will try to tune the Oracle configuration,
and the last, I will try the I/O part
so, I will know how much each phase take effects.

> If you know your subsystem has cache then reduce the amount of system buffers
from sysdef today, my bufpages value is 198371 (yesterday 416197)
from my kmtune, the value of bufpages is 0 (zero = flexibel?)
based on system performance tuning from http://docs.hp.com/en/64/hppg/hpconf.htm,
the result is "buffer cache is too small" (from chapter 6.2.2 point 2)
it makes me a little bit confuse, because I think the value is quite large.

> values of maxusers... how many concurrent users?
maxusers=32
my users=20-25
I will uplift this value upto 256, because the weblogic recommends this value
for my HP-UX server (and just in case, a user opens more than 1 session)

> I there a good reason for maxuprc 3686?
no idea yet.. I'll try to find out.

> I would reduce of 200000 your redo and data buffers
> Can you give this a try and lets see what happens?
Ok. I will try this at my second phase of tuning (oracle tuning) and give you information
about the result

At first phase, I will change these kernel configuration :
max_thread_proc 256 upto 1024 (weblogic recommendation for my version of HP-UX)
maxfiles 60 upto 256 (weblogic recommendation for my version of HP-UX)
maxusers 32 upto 256 (weblogic recommendation for my version of HP-UX)
nkthread 7184 - (weblogic recommends 3635 for my version of HP-UX)
nproc 4096 - (weblogic recommends 2068 for my version of HP-UX)
ncallout 7200 - (weblogic recommends 2084 for my version of HP-UX)
maxuprc 3686 downto 2048 (trial)

I will try the Unix Kernel tune today, and I will give you the result tommorow.

Thanks.

Dont forget :
dbc_max_pct 8
dbc_min_pct 5
swapmem_on 0

The idea is to stop the box from swapping to disk, then try to see what you have when loaded, and if then, you still have enough free mem, then see if we could increas agin oracle SGA and see if it benefits in terms of performance...

LVM:
vg00 isnt what we are interested in but lines like:
/dev/vg01/lvol1 /<mydata> vxfs rw,suid,largefiles,delaylog,datainlog 0 2

Here we learn you have largefiles enabled...

Back later... sorry...

> Dont forget :
> dbc_max_pct 8
> dbc_min_pct 5
> swapmem_on 0
ok.. I will try this later, because my partner wants to try to configure these one first :

max_thread_proc from 256 upto 1024
maxfiles from 60 upto 256
maxusers from 32 upto 256
nproc 4096 downto 2068

let's see what hapens in the next few days first.

keep us informed then, I would be the first surprised if those modifications impove things so Im very interested in knowing... mind you its your pal loosing his time in reboot not me...
About hte disks:
ioscan -funC disk
ioscan -funC fc
vgdisplay -v vg01 but just the last part like:

 --- Physical volumes ---
   PV Name                     /dev/dsk/c14t1d0
   PV Name                     /dev/dsk/c15t1d0 Alternate Link
   PV Status                   available                
   Total PE                    3781    
   Free PE                     523     
   Autoswitch                  On        
   Proactive Polling           On               

   PV Name                     /dev/dsk/c14t1d2
   PV Name                     /dev/dsk/c15t1d2 Alternate Link
   PV Status                   available                
   Total PE                    3781    
   Free PE                     523     
   Autoswitch                  On        
   Proactive Polling           On               

   PV Name                     /dev/dsk/c14t1d4
   PV Name                     /dev/dsk/c15t1d4 Alternate Link
   PV Status                   available                
   Total PE                    3781    
   Free PE                     571     
   Autoswitch                  On        
   Proactive Polling           On               
...
   --- Physical volume groups ---
   PVG Name                    pvg_es03a                  
   PV Name                     /dev/dsk/c14t1d0           
   PV Name                     /dev/dsk/c15t1d1           
   PV Name                     /dev/dsk/c14t1d2           
   PV Name                     /dev/dsk/c15t1d3           

   PVG Name                    pvg_es03b                  
   PV Name                     /dev/dsk/c14t1d4           
   PV Name                     /dev/dsk/c15t1d5           
   PV Name                     /dev/dsk/c14t1d6           
   PV Name                     /dev/dsk/c15t1d7           

hi..
those modifications does not effect so much.

i will try the : dbc_max_pct, dbc_min_pct and swapmem_on

> ioscan -funC disk
Class I H/W Path Driver S/W State H/W Type Description

disk 0 1/0/0/3/0.6.0 sdisk CLAIMED DEVICE HP 73.4GST373453LC
/dev/dsk/c0t6d0 /dev/rdsk/c0t6d0
disk 1 1/0/0/3/1.2.0 sdisk CLAIMED DEVICE DVD
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 2 1/0/1/1/0/1/1.6.0 sdisk CLAIMED DEVICE HP 73.4GST373453LC
/dev/dsk/c3t6d0 /dev/rdsk/c3t6d0
disk 37 1/0/2/1/0.1.0.0.0.0.0 sdisk CLAIMED DEVICE COMPAQ MSA1000 VOLUME
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0

> ioscan -funC fc
Class I H/W Path Driver S/W State H/W Type Description

fc 0 1/0/2/1/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter

> vgdisplay -v vg01
--- Physical volumes ---
PV Name /dev/dsk/c4t0d0
PV Status available
Total PE 13022
Free PE 22
Autoswitch On

what kind of box is this? (L class? or RPXXXX that I dont know? cmd: model will tell you...)
From what you just posted I guess vg00 is mirrored...
Is this a production box? because I have some concerns about vg01...(only one fc? and one spindle...)

dbc_min_pct 6 now!
I suggest you leave in system some formulas so you just have to change one param and it will affect the others:
max_thread_proc ((MAXUSERS4)+256)
maxdsiz 2063835136
maxdsiz_64bit 0X80000000
maxfiles 2048
maxfiles_lim 4096
maxssiz 0X8000000
maxssiz_64bit 0X40000000
maxswapchunks 9600
maxtsiz 0X8000000
maxtsiz_64bit 0X40000000
maxuprc (NPROC6/9)
maxusers 300
maxvgs 20
msgmni 100
msgseg 20000
msgssz 400
msgtql (NPROC)
nclist (200+16MAXUSERS)
nfile (15NPROC+2048)
nflocks (NPROC)
nkthread (((NPROC9)/4)+512)
nproc (512+9MAXUSERS)
npty 256
nstrpty 72
nstrtel 200
semmni (NPROC)
semmns (SEMMNI*2)
semmnu (NPROC-4)
semume 30
shmmax 0X40000000
shmmni 300
shmseg 150

and of course:
dbc_max_pct 8
dbc_min_pct 6
swapmem_on 0