I have a customers that is getting grid alerts that swap is over 95% utilized. When I do swap -l on the machine I get the following results.
$ swap -l
swapfile dev swaplo blocks free
/swap/swapfile - 16 6291440 6291440
/swap/swapfile2 - 16 8191984 8191984
Solaris is not my strongest OS to work with but it looks to me that no swap is being used. Can anyone give me some direction?
Thanks, Michael
Looks like you're not using swap space at all. Do you have root access? If so, run:
echo ::memstat | mdb -k
# echo ::memstat | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 635150 4962 15%
Anon 1518806 11865 36%
Exec and libs 16421 128 0%
Page cache 14143 110 0%
Free (cachelist) 10962 85 0%
Free (freelist) 1983895 15499 47%
Total 4179377 32651
Physical 4112976 32132
How are customers getting alerted. Do customers have their own virtualized versions?
Oracle Enterprise Manager grid alerts.
Well, you're not out of RAM, but that doesn't mean you're not "out" of swap.
Solaris does not do memory overcommit. So if a process does a malloc() call for 16 GB, the OS will reserve 16 GB of swap for that process. Even if the process never uses any of that 16 GB so that it shows up as allocated memory in the mdb ::memstat output above, it still gets reserved. Even if you've never used it.
And if that process calls fork(), the child process will get another 16 GB of swap reserved. Or not - if it's not there, the fork() could fail (I don't remember exactly what happens offhand). Or, you could wind up seeing your swap error message.
Are you seeing any types of failures in application logs, or /var/adm/messages? Generally, if you really are running out of swap you should see something there.
What's the output of "prstat -t"?
NPROC USERNAME SIZE RSS MEMORY TIME CPU
92 oracle 1514G 919G 100% 119:39:11 6.7%
41 root 201M 112M 0.0% 5:44:39 0.0%
1 smmsp 7744K 2656K 0.0% 0:00:23 0.0%
6 daemon 84M 76M 0.0% 0:03:07 0.0%
Looks like the Solaris 10 version of prstat doesn't show swap reservations.
Try this dtrace script:
This customer is running Solaris 5.10.
$ su -
Password:
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
#
AKA Solaris 10.
Ah, no wonder I felt lost.
# ./swapinfo.d
RAM _______Total 32760 MB
RAM Unusable 108 MB
RAM Kernel 17454 MB
RAM Locked 10426 MB
RAM Used 17592186034039 MB
RAM Free 15147 MB
Disk _______Total 7071 MB
Disk Resv 7071 MB
Disk Avail 0 MB
Swap _______Total 7761 MB
Swap Resv 7071 MB
Swap Avail 689 MB
Swap (Minfree) 4081 MB
Something is definitely reserving swap space, the question is what. What does "prstat -s size" show? Then you can use "pmap" on the big ones and hopefully find which one(s) are mapping a lot of anonymous memory.
Another possibility are shared memory segments. There are ways to create those that don't use any swap space because they get locked into memory. I know DISM segments do use swap because those aren't locked into memory like plain ISM segments.
---------- Post updated at 06:09 PM ---------- Previous update was at 06:06 PM ----------
And FWIW, if you do "uname -a" you'll see "SunOS ... 5.10 ...". Solaris replaced SunOS 4.x, so it's SunOS 5.x. Solaris 8 would be SunOS 5.8. Solaris 11 is SunOS 5.11.
Looks like it's all Oralce
# prstat -s size
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
8429 oracle 18G 11G sleep 59 0 0:21:42 0.0% oracle/1
647 oracle 18G 11G sleep 59 0 0:12:12 0.0% oracle/1
445 oracle 18G 11G sleep 59 0 0:01:42 0.0% oracle/1
11105 oracle 18G 11G sleep 59 0 0:00:15 0.0% oracle/1
29643 oracle 18G 11G sleep 59 0 4:04:21 0.0% oracle/13
2501 oracle 18G 11G sleep 59 0 2:49:11 0.0% oracle/14
5893 oracle 18G 11G sleep 59 0 12:32:39 0.1% oracle/1
20100 oracle 18G 11G sleep 59 0 0:16:58 0.0% oracle/1
2445 oracle 18G 11G sleep 59 0 1:25:07 0.0% oracle/1
21346 oracle 18G 11G sleep 59 0 0:04:08 0.0% oracle/1
20476 oracle 18G 11G sleep 59 0 0:08:30 0.0% oracle/1
26541 oracle 18G 11G sleep 59 0 0:04:53 0.0% oracle/1
16517 oracle 18G 11G sleep 59 0 0:20:39 0.0% oracle/15
20308 oracle 18G 11G sleep 59 0 0:01:22 0.0% oracle/1
26438 oracle 18G 11G sleep 59 0 0:05:29 0.0% oracle/1
20219 oracle 18G 11G sleep 59 0 0:06:38 0.0% oracle/1
20641 oracle 18G 11G sleep 59 0 0:07:01 0.0% oracle/1
19368 oracle 18G 11G sleep 59 0 0:06:40 0.0% oracle/1
26784 oracle 18G 11G sleep 59 0 0:01:51 0.0% oracle/1
16515 oracle 18G 11G sleep 59 0 0:14:09 0.0% oracle/258
16513 oracle 18G 11G sleep 59 0 0:14:06 0.0% oracle/258
22948 oracle 18G 11G sleep 59 0 0:02:42 0.0% oracle/1
22777 oracle 18G 11G sleep 59 0 0:01:46 0.0% oracle/1
27393 oracle 18G 11G sleep 59 0 0:03:24 0.0% oracle/1
19893 oracle 18G 11G sleep 59 0 0:01:44 0.0% oracle/1
19384 oracle 18G 11G sleep 59 0 0:00:47 0.0% oracle/1
27196 oracle 18G 11G sleep 59 0 0:01:51 0.0% oracle/11
27161 oracle 18G 11G sleep 59 0 0:00:46 0.0% oracle/1
21741 oracle 18G 11G sleep 59 0 0:12:29 0.0% oracle/1
23389 oracle 18G 11G sleep 59 0 0:00:25 0.0% oracle/1
16519 oracle 18G 11G sleep 59 0 0:30:03 0.0% oracle/18
29731 oracle 18G 11G sleep 59 0 0:19:17 0.0% oracle/1
Total: 135 processes, 877 lwps, load averages: 1.08, 1.33, 1.64
---------- Post updated 06-27-14 at 02:00 PM ---------- Previous update was 06-26-14 at 06:25 PM ----------
Looking at this data again... Should more swap be added? Is this telling me that only 7761 MB of swap exists?
What does this show:
swap -s
# swap -s
total: 12165728k bytes allocated + 7414976k reserved = 19580704k used, 697584k available
Based on this output, it seems that you have only 3% of free swap space:
697584 / (19580704 + 697584) 3.4%
Yes, that is what this server is complaining about.
It's oracle. We have that problem any time a dba resizes SGA. That 100% usage in prstat is your clue.
Mount another new disk to the system. Use mkfile to create about about 400-500GB of swap space on the device. Rule of thumb with oracle: 2xRAM size == total swap space. Add the the swap with the swap -a command. This is sketchy on details, but if you need exact advice try:
For a direct attach disk, the example does not use mkfile, rather format:
UtahSysAdmin.com | How to add a swap drive to Solaris 10
No reboot required.
If you are using a SAN you can add a new lun and do the above. zfs is slightly different. Ask back for more help.
I wish I could remember enough about the Oracle installations I've worked on to know if we didn't really need to do that. If the SGA is configured to use intimate shared memory (ISM), then it won't require any swap reservation because ISM is locked into RAM and can't be swapped out.
Note that you do NOT want to use DISM - that does require swap reservation.