The environment we're using is a T5440 running Solaris 10 LDOMs and zones configured within the LDOM. The LDOM has 32Gb of RAM and 32Gb of swap.
Last week, we had an issue where the MQ server on zone 3 crashed as it ran out of swap, apparently caused by zone 2 usage. I understand that RAM and swap are shared across the zones so that's understandable. The real issue appears to be that we're using up swap even though there is plenty of RAM still available. I'm not the SA, but I understand that there is no specific resource capping in place.
3 questions...
1) Has anyone come across this scenario before with swap being used, seemingly in preference to RAM!
2) What's the best way to determine what RAM and SWAP is being used by which processes, and can this be tracked dynamically? Would any tracking need to occur at the global level?
3) If we can determine a rogue process, can dtrace or similar be used to delve deeper?
You are misunderstanding the issue. RAM and swap are related but different beasts. You can certainly exhaust your swap (a.k.a. virtual memory) while RAM is still available. Solaris doesn't overcommit memory so if you haven't swap enough to back memory reservations, you'll get what you observe.
"prstat -Z" and "swap -s" will help you investigating the issue.
Clearly I am misunderstanding! I thought that RAM was used in preference, and when there wasn't enough available, the OS started swapping memory pages to disk. If this isn't the case, I'd appreciate a one paragraph tutorial on what really happens!
Most UNIX systems will attempt to guarantee there's enough swap available for all allocated memory, instantiated or not. Things that allocate masses of memory but don't ever use it can exhaust all available swap space even though they never had any physical memory pages assigned to them.
(Linux systems are a bit unusual in generally having "virtual" swap, in which swap doesn't get assigned to allocated memory until it needs to be paged out. It will allow far more memory to be allocated than the system has swap, but if the processes in memory actually use more memory than it has mem + swap combined, processes will start being killed to reap memory.)
That statement is correct but is unrelated to what you experience. As Corona688 already explained, the issue isn't with usage but reservation. Adding enough swap space for all reservations (malloc) to fit will fix the issue unless the is a memory leak.
So, for example, I start a Java process that the OS knows may require 1Gb of space at some point in its life. Is the above statement saying that 1Gb of swap will be allocated and therefore unavailable for use by any other processes, but initially only, say, 100Mb will actually be utilised? And if I had 10 such Java processes, I've taken up a huge proportion of my available swap. However, as I've got 32Gb of physical RAM, the swap will never actually be utilised!?
Finally, the prstat -Z gives me SIZE and RSS values, and for each zone a SWAP, RSS and MEMORY column. What do these values represent?
It's not that the OS knows it will require 1GB, it's that Java asked for 1GB. (Though some systems will reserve more than the program asks for anyway.) Otherwise, yes. This is to guarantee that even worst-case, when everything on the system decides to use all memory that's been allocated to it, the system actually has enough to do so.
SWAP is telling the amount of swap (under Solaris terminology: virtual memory which includes both the swap area and a large part of the RAM) used or allocated in that zone.
RSS is telling how much RAM is used, MEMORY is the share of total RAM used by the zone.
I thought that RAM was used in preference, and when there wasn't enough available, the OS started swapping memory pages to disk. If this isn't the case, I'd appreciate a one paragraph tutorial on what really happens!