Page cache too large

UNIX Solaris
1

I have a solaris10 box running a java application on it. Whenever the java app is used heavily, the amount of free memory decreases fairly rapidly. I believe I have eliminated the running applications from the culprit list. In the process I have found that the page cache is consuming about 20G of the 24G of memory on the box. I read in other posts that this is okay, since the cache is easily dumped when required by applications. But after the box reaches around 1% free memory, swap starts filling up. This makes monitoring the amount of free memory and swap pretty much useless.

Any ideas on how to identify what is getting cached, and/or how to keep it down to something sensible ?

echo "::memstat"|mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     208042              1625    7%
Anon                       200571              1566    6%
Exec and libs                2644                20    0%
Page cache                2452919             19163   79%
Free (cachelist)           111729               872    4%
Free (freelist)            138338              1080    4%

Total                     3114243             24330
Physical                  3059418             23901

last pid:  6927;  load avg:  0.44,  0.56,  0.56;       up 204+14:35:29                                                               13:55:18
74 processes: 72 sleeping, 2 on cpu
CPU states: 90.1% idle,  6.9% user,  2.9% kernel,  0.0% iowait,  0.0% swap
Memory: 24G phys mem, 1539M free mem, 24G total swap, 21G free swap

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  9529 root      33  59    0 1094M  586M sleep   18:06  0.02% java
 12190 root      68  44    0  505M  472M cpu/3   72.9H  8.02% java
   551 daemon     4  59    0  389M  333M sleep    5:18  0.00% nfsmapid
  2935 noaccess  27  59    0  220M   56M sleep  180:44  0.01% java
 19134 named      7  59    0   44M   18M sleep   29:57  0.01% named
   789 root      16  59    0   14M 2640K sleep    2:32  0.00% fmd
     7 root      13  59    0   12M 4056K sleep    2:50  0.00% svc.startd
 26079 root       1  59    0   11M 4976K sleep    0:00  0.00% sshd
2

Could it be a memory leak in either the Java app or in Java itself?

How to Fix Memory Leaks in Java | Javalobby

Handling memory leaks in Java programs

3

If the java app was leaking memory, wouldn't it's memory usage grow over time ?

Also, killing and restarting the application does not clear the memory, I have to reboot the box to free up the memory.

In the meantime, during a maintenance window I will take a heap dump and try to start debugging it. Thanks for the tips!

--
Tim

4

The issue isn't related to your java application leaking heap memory.

What I suspect is the java application creating huge files that would stay the /tmp directory. The "Page cache" is reporting tmpfs usage, beyond other things.

What says: "df -k /tmp" ?

1 Like
5

Wow, you sliced that with occam's razor for sure!
It was a stupid file that was being logged to in /tmp/. Which of course, was cleared out on a reboot.

Thanks so much.

--
Tim