We have a server at a client site running AIX 5.3, which we just up the RAM to 32GB, from initially 16GB (if I'm not mistaken).
This server is our Application server running J2EE applications on top of Oracle Internet Application Server. Recently we encountered one of the batch jobs hitting out of memory error.
From NMON I see that on average, there's 5GB of free memory available. %MAXPERM is set to '15' and %MINPERM is '5'. lru_file_repage is '0'. %comp is around '70' and %numperm is '15'.
I would like to ask
a) after adding real memory to the server, is there any additional changes we need to do to the VMO?
b) is the %MAXPERM too low?
a)
Usually not - many parameters regarding memory are percentage based. Others have fixed values. When doubling the memory it should usually run smoother than before if there was a memory shortage or no tuning.
b)
Not easy to say this way. When using lru_file_repage=0, it is recommended to have maxperm% back to 80 which is default and depending on memory usage, some do even set 85 or 90.
A box running out of memory will usually start to page out, before anything crashes, and sowith become very slow. I heared once that Java dislikes to be paged out - not sure if this is true or still a problem though. It could also be that you can increase memory usage inside your application server (I am not very familiar with java but heapsize stuff etc. come to my mind), especially after having added real memory.
Post a vmstat -vs and a svmon -G . It could also be, that the ulimits for the user where this application is running, has too tight limits. A ulimit -a issued from that user will show the current limits.
01:37:30:911 java.lang.OutOfMemoryError
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java(Compiled Code))
at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java(Compiled Code))
at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java(Compiled Code))
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java(Compiled Code))
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java(Compiled Code))
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java(Compiled Code))
at sqlj.runtime.ExecutionContext$OracleContext.oracleExecuteQuery(ExecutionContext.java(Inlined Compiled Code))
---------- Post updated at 11:39 AM ---------- Previous update was at 11:36 AM ----------
@zaxxon: below are the output of the commands you suggested me to run
$vmstat -vs
6773583119 total address trans. faults
85402374 page ins
179082270 page outs
0 paging space page ins
0 paging space page outs
0 total reclaims
1568460598 zero filled pages faults
437557684 executable filled pages faults
1598879605 pages examined by clock
247 revolutions of the clock hand
176110235 pages freed by the clock
29164084 backtracks
1067 free frame waits
0 extend XPT waits
6849007 pending I/O waits
264484781 start I/Os
56911924 iodones
43584605859 cpu context switches
15545396044 device interrupts
1335402576 software interrupts
3604933076 decrementer interrupts
212611 mpc-sent interrupts
212611 mpc-receive interrupts
19173376 phantom interrupts
0 traps
101258980092 syscalls
8134656 memory pages
7786454 lruable pages
1862642 free pages
1 memory pools
1772455 pinned pages
80.0 maxpin percentage
5.0 minperm percentage
15.0 maxperm percentage
14.9 numperm percentage
1166944 file pages
0.0 compressed percentage
0 compressed pages
14.9 numclient percentage
15.0 maxclient percentage
1166944 client pages
0 remote pageouts scheduled
19863 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
2228 filesystem I/Os blocked with no fsbuf
47 client filesystem I/Os blocked with no fsbuf
8856 external pager filesystem I/Os blocked with no fsbuf
$ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) 32768
coredump(blocks) unlimited
nofiles(descriptors) 2000
#svmon -G
size inuse free pin virtual
memory 8134656 6435521 1699135 1772857 5254449
pg space 3276800 13467
work pers clnt other
pin 1499599 0 0 273258
in use 5254449 0 1181072
PageSize PoolSize inuse pgsp pin virtual
s 4 KB - 6315233 13467 1674265 5134161
m 64 KB - 7518 0 6162 7518
This looks like a Java memory error. Maybe check with your application server if you can set parameters for memory pools/heapsize (whatever there is and they are called) etc. to a higher value. This does not look like a problem with the OS.
Hmm... Is there a reason why memory usage for the application is limited?
Looking at the other data it seems to be quite ok, but the tuning parameters would be interesting to know to effectively assess the situation. Please post the output of "vmo -a", "ioo -a" and "schedo -a".
The numbers in the "vmstat -vs" output are collected since last reboot. What was the uptime of the system when you issued the command? Would it be possible to reboot the system to start over these statistics?
Good point - I should sometimes read what my eyes see If possible and nothing else running on this box, set this for the user to -1 (unlimited) and logoff/logon and restart the application with the new limits. That could be the problem indeed.
is your java app potentially 32bit like websphere or certain other apps. Than you might run into the trap with limited shared memory as this is not unlimited like for 64bit apps. To work around this, put export EXTSHM=ON in the profile of the user which runs the java application.
Further several Java apps are only allowed to consume as much (shared) memory as explicitely is allowed within the application configuration.
And yes your memory settings do not allow the system to use the memory it has. Try the settings we keep recommending -
try to mount your filesystems with noatime and consider carefully if you need to really keep things in memory - if not consider mounting the corresponding filesystem rbrw which sends back the memory into the free list as soon as the IO is completed.
Kind regards
zxmaus
eehhhmm.. that means that for example when i execute a java process without defining the heap size, the maximum size the heap will grow is 32MB? Once the heap reaches 32MB, then it will exit with an 'OutOfMemory' error. unless I set the ulimit memory to 'unlimited' then it will exceed 32MB, right?
I am trying to understand what the ulimit memory is.
Ok, thanks everyone.. the first thing to do now is to set the ulimit for memory to 'unlimited'. Will monitor for awhile and see if it's ok..
---------- Post updated at 02:54 PM ---------- Previous update was at 02:30 PM ----------
Ohhhkay.. i am a bit confused now..
when I am login as a normal user, e.g 'oracle' and execute 'ulimit -a', i see the memory is 32768 Kb. But when i view the /etc/security/limits file, 'oracle' user has the following limits set:
new ulimits only become active if you logout/login again to the box as this user.
If you have oracle running, this would include a restart of oracle
apart from this - you are setting only soft limits which may or may not be limited by your hard limits - so if you really want unlimited memory than you need to amend your hard limits as well - so you may want to have a setting like the below in your /etc/security/limits:
There is a file "/etc/security/limits" which steers all the ulimits. It holds all the limits in the form of stanzas. There is a "default"-stanza which values will be used if no specific value is assigned for a user.
You can change the settings via the "chuser" command (for another user) or the "ulimit" command (from within the respective user account) or even directly (as root) using a text editor.
For more information about /etc/security/limits refer to the man page accessible via man limits .
If possible and nothing else running on this box, set this for the user to -1 (unlimited) and logoff/logon and restart the application with the new limits. That could be the problem indeed.