Problem with Swap consumption

Hi Experts,

I have M4000 server with 132 GB Physical memory. 4 sparse zones are running under this server, which are running multiple applications. I am not getting any pointer, where swap space is getting consumed. Almost 97% of swap space is being used. I checked all /tmp (of zones as well), which are almost blank.

# zfs list | grep -i swap
rpool/swap 4.05G 60.5G 4.05G -
# swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 256,1 16 8388592 8157216
# swap -s
total: 53587272k bytes allocated + 19471528k reserved = 73058800k used, 1925368k available
# vmstat 2 2
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr lf s0 s1 s2 in sy cs us sy id
2 0 0 4122144 31798056 75 328 0 0 0 0 0 0 87 86 -0 16897 60434 24387 10 6 84
0 0 0 1924984 30637104 28 54 0 0 0 0 0 0 165 165 0 15363 11150 17851 2 3 95
# df -k | grep -i swap
swap 1934264 480 1933784 1% /etc/svc/volatile
swap 524288 15472 508816 3% /tmp
swap 1933920 136 1933784 1% /var/run
# prstat -s size
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
12628 rmdadmin 1999M 941M sleep 29 10 0:01:31 0.0% java/86
17905 rmdadmin 1946M 669M sleep 29 10 2:18:05 0.0% java/84
6141 tomsadmd 1499M 1356M sleep 59 0 9:34:34 0.0% BIService/1622
4148 lsradmin 1427M 1401M sleep 29 10 1:16:38 0.0% BIService/43
7116 35810 1389M 825M sleep 29 10 14:06:38 0.1% java/151
7097 35810 1351M 705M sleep 29 10 12:39:33 0.2% java/149
19028 35810 1335M 1023M sleep 29 10 6:30:59 0.0% java/71
16462 sys11adm 1298M 1087M sleep 59 0 2:24:41 0.0% java/122
11745 mfradmd 1212M 103M sleep 59 0 0:14:28 0.0% java/24
17 sys11adm 1185M 963M sleep 47 4 1:24:11 0.0% java/57
16529 sys11adm 1180M 318M sleep 59 0 0:36:09 0.0% java/39
17863 rmdadmin 1171M 451M sleep 59 0 1:15:28 0.0% java/61
16436 sys11adm 1154M 373M sleep 59 0 11:05:52 0.0% java/66
41 sys11adm 1141M 311M sleep 47 4 0:33:32 0.0% java/39
22658 35810 1132M 823M sleep 29 10 8:17:06 0.0% java/93
6106 tomsadmd 1062M 745M sleep 59 0 2:44:37 0.0% java/1796
21044 apsadmsy 1033M 305M sleep 59 0 76:27:58 0.0% java/199
23037 24943 976M 814M sleep 59 0 3:05:00 0.0% java/75
19257 obieed 965M 641M sleep 47 4 9:32:09 0.0% java/44
14442 24943 930M 838M sleep 59 0 0:19:57 0.0% BIService/21
831 24943 858M 665M sleep 59 0 31:20:25 0.0% java/39
21096 apsadmsy 846M 420M sleep 29 10 4:40:07 0.0% java/79
19029 35810 827M 651M sleep 29 10 3:39:34 0.0% java/45
22659 35810 815M 641M sleep 29 10 3:29:17 0.0% java/44
7947 35810 797M 468M sleep 29 10 1:23:46 0.0% java/39
852 tomsadmd 794M 610M sleep 59 0 6:07:22 0.0% java/77
8884 vdwadmd 793M 587M sleep 59 0 6:31:07 0.0% java/108
7946 35810 793M 459M sleep 29 10 1:22:50 0.0% java/39
12636 lssadmd 792M 516M sleep 59 0 8:33:14 0.0% java/80
6055 weradmd 757M 607M sleep 59 0 2:47:18 0.0% java/90
17324 24943 734M 353M sleep 59 0 2:25:37 0.0% java/42
20217 lssadmd 730M 530M sleep 59 0 42:02:53 0.1% java/104
25202 sys9admd 723M 580M sleep 59 0 1:09:51 0.0% java/144
28867 hpsuadmd 712M 374M sleep 59 0 11:44:01 0.0% java/67
21152 apsadmsy 708M 368M sleep 29 10 3:52:47 0.0% java/65
19764 hpsuadmd 701M 502M sleep 59 0 60:57:07 0.0% java/58
15493 24943 700M 472M sleep 59 0 3:19:35 0.0% java/91
23111 lssadmd 690M 381M sleep 59 0 6:07:23 0.0% java/43
27861 fpradmin 689M 564M sleep 59 0 3:26:04 0.0% java/96
26187 sppsadmd 685M 520M sleep 59 0 3:44:05 0.0% java/69
27870 fpradmin 670M 581M sleep 59 0 3:35:13 0.0% BIService/20
27162 56022 664M 596M sleep 59 0 1:11:02 0.0% java/75
Total: 688 processes, 20735 lwps, load averag

Please help me, as it is impacting my productioni server.

Regards
Abhishek Singh

I was under the impression that all memory had to be backed by swap on most UNIX systems...

It does not look like any of the virtual filesystems like TMPFS which can use up swap space are a problem.

However long running Java applications has a habit of slowly eating up memory and swap. How long since this system was rebooted?

The entire swap partition is treated as a single global resource to processes running in both global and non-global zones. If you are running Solaris 10 8/07 or later, you can limit the amount of swap on a per-zone basis using the 'swap' property of the capped-memory resource to cap the amount of virtual memory (VM) that a zone uses. This can also be set dynamically with the resource control zone.max-swap (check if this is correct name.) Have you tried setting a cap on swap space usage per zone?

fpmurphy,
Thanks for your prompt reply.
Server is not rebooted since long time (over a year). Here is the config for four zones -

 
Zone1 - 
capped-memory:
physical: 6G
[swap: 12G]
[locked: 6G]
rctl:
name: zone.max-swap
value: (priv=privileged,limit=12884901888,action=deny)
rctl:
name: zone.max-locked-memory
value: (priv=privileged,limit=6442450944,action=deny)
 
Zone2 -
No capped memory or swap dedicated
 
Zone3 -
capped-memory:
physical: 4G
 
Zone4 - 
capped-memory:
physical: 4G

We can set up capped swap space for other zones also, but I feel 97% utilization is too high, so I was concerned, if something is eating up too much.

Regards

Try to use dtrace to see what is eating up your swap space. You can find some pretty good dtrace scripts here that will help you figure it out.

Top Ten DTrace (D) Scripts

Is there a problem with processes not getting the memory they request? If not, it's probably just that the page cache has grown over the extended uptime.

Thanks all for the help.
sboots, I was able to see that there is no problem with any process. It is just extended uptime (428 days), which might have consumed mem and cpu as achenle said. I will talk to client to reboot the box.
Thanks again for your expertise.

This is hopefully no more the case with modern Unixes, and especially Solaris since a couple of decades.

---------- Post updated at 23:39 ---------- Previous update was at 23:36 ----------

The page cache isn't accounted as swap (virtual memory) usage. That wouldn't make sense to cache disk blocks with other disk blocks anyway. It isn't either accounted as RAM usage, unless you use ZFS.

---------- Post updated at 23:43 ---------- Previous update was at 23:39 ----------

Please post these commands output:

echo ::memstat | mdb -k
prstat -Z

Unless you have a memory leak in an application, uptime doesn't use memory.

Why not simply adding some swap space instead of rebooting ?

---------- Post updated 17-03-11 at 11:43 ---------- Previous update was 16-03-11 at 23:43 ----------

I would on the opposite state that java applications are unlike to eat up memory/swap in the long term as their heap size is bounded at startup by design. C applications have no such limitation so can leak memory until their VM space is exhausted (or sooner).