Coredumps and swap - was part of Solaris Mem Consumption

We have Sun OS running on spark :

SunOS ciniwnpr67 5.10 Generic_118833-24 sun4u sparc SUNW,Sun-Fire-V440

Having Physical RAM :

Sol10box # prtconf | grep Mem
Memory size: 8192 Megabytes

My Top Output is :

130 processes: 129 sleeping, 1 on cpu
CPU states: 98.8% idle, 0.2% user, 1.0% kernel, 0.0% iowait, 0.0% swap
Memory: 8192M real, 343M free, 2282M swap in use, 2195M swap free

Which shows about 7 gigs of RAM already consumed.

My sar -k shows :

SunOS ciniwnpr67 5.10 Generic_118833-24 sun4u 08/07/2008

07:29:53 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail
07:29:54 4184032128 4044621477 0 901931008 621211704 1 1723777024 0

And if i under stood it right then i have around 4044621477 + 621211704 + 1723777024 Bytes of memory or 4.5 gigs used by kernel itself ??

If it is true then it i think some thing is wrong with my solaris installtion Any suggestions pls ?

:pAnd not to forget, as a rule of thumb, swap device should be configured double the size than of your physical memory. (16384)
V440 is a stable server. Yours should keep going.

well here is the issue ..:

As per Top i am using 7 gigs of RAm.. now the moment i run some thing else then the current applications it page outs increases significantly from almot "0" to 4ooo +

That is causing the new processs to take more time ..

check this out
How to Check Kernel Memory Allocation (sar -k) (System Administration Guide: Advanced Administration) - Sun Microsystems
do this:
# prtmem
#netstat -kp | grep pp_kernel
#mdb -k

here is the thing :

# prtmem - is a part of memtool and that is not recomended to run on prod i guess

#netstat -kp | grep pp_kernel
For me there is no option as - k in netstat on solaris 10

#mdb -k

I really dont have any idea about this dubugger but i will chk it out ..

Tell me this thoguh what sar -k shows in my case .. didnt that meant that kernel is using so much memory ?

That is a very old rule of thumb which I no longer agree with. Memory is so cheap and plentiful these days that you should hardly ever require swap, if you do it is a sign of problems. I generally don't ever configure more than 4GB of swap, even on a 64+GB system.

In fact, if your physical memory is alot, of course Im not asking you to give 128 GB for your swap. But you will t least need 16Gb of swap for a 64 0r 128 configuration minimally.

This is the architecture of Solaris. You will need additional swap for your paging and the space for your coredump(which might be Huge in size). You won't want your system to crash further after a system panic, would you?:wink:

incredible - of course your dump could be rather large but i run m9000's and have never seen a core file greater then a few gigs tops. we are talking about 1-4 TBs of memory and 64 sparc64 VI (128 cores) processors.

For your info M series is a collaboration of Fujitsu with SUN and their architectures do differ.. Though you may find M9000 is working like a 6900 system.. :stuck_out_tongue:
If its gonna be a "SUN" mid-range or highend servers, what I say still makes sense.

Yes, I agree with this. The old 2 pounds of swap for every pound of memory was back in the old days when memory was expensive and it was cheaper to use disk swap.

If you have 7 GB of RAM, something is really wrong if you need 14GB of swap.

4GB of swap is more than enough, good post Annihilannic. Thanks.

There is a lot of infomation in this thread which is not valid. Almost everything Incredible has stated in this thread is incorrect in terms both of system configurations and reasons for choosing swap size.

A few points worth note.

  1. You don't need to allocate any swap space to deal with savecores, and have not since Sorlais 8.
  2. Solaris will never go into a panic-reboot cycle as a result of not having savecore space. It will simply not save a core dump if it has no space.
  3. Twice memory as swap is no longer a good choice unless you really can't afford to upgrade.
  4. If you have a lot of pagout you do not have enough memory, it's as simple as that.
  5. You do not need minimum 16GB swap for 64 or 128GB of memory, but you may need to have more swap if you have applications using ISM (Intimate shared memory) or DISM (Dynamic Intimate Shared Memory) such as Sybase or Oracle databases.
  6. There is no reason to treat an M-Series differently from any other Solaris box.

In summary you can get by to some extent if you don't have enough memory by adding swap, but it will hurt performance. Ideally you should have enough memory to run all your applications in memory, and the general rule of thumb nowadays is about 30% of memory for swap but there are more detailed reccomendations in the Solaris documentation.

Jim Laurent at Sun wrote a blog on this topic about a year ago, which you could look up.

EDIT: Found the blog:Solaris FAQ: Myths and facts about Solaris swap space : Jim Laurent's Weblog

Excellent followup reborg. Thanks for taking the time to set the record straight.

I'm afraid this statement is misleading.
Solaris still requires a single dump slice large enough for kernel crash dump data to be recorded, and there is no much point not using a swap area for it.
Even while this data is compressed since Solaris 8, I wouldn't recommend to use less than the RAM size as swap size as it is the only way to guarantee a crash dump will fit.

Most folks seem to say that 4GB is large enough for any Solaris core dump. In addition, you can specify to limit the size of the core dump.

Since I use Linux, I don't recall ever needing a core dump, and normally I find them a waste of disk space on most systems. Better to have a kernel that does not dump cores .... :smiley: that you have massive core dumps.

In other words, I don't think that core dumps should be must of a factor in thinking about swap.

You cannot limit the kernel dump size so I'm afraid you are confusing processes core dumps and kernel crash dumps.

Anyway, my point was more about this misconception about kernel dumps that wouldn't require a dump device while they still do.

Whether you run Linux, Solaris or whatever OS, kernel crashes do happen. When you run business critical applications on multi-million dollar machines with dozens or hundred of gigabytes of RAM, you don't want to save a few gigabytes of disk space knowing this prevents analyzing what caused the crash.

It's the same reasoning why you do not remove the black-box from a commercial plane to save weight ...

I am not sure sure you are correct.

Solaris has a kernel tuning parameter called:

dump_cnt  - limits the size of the dump

So, even you are correct and dump_cnt only limits the size of the process core dumps (btw, it does not say this in the docs), kernels are generally much smaller.

Your argument about "airplanes and black boxes" is a form of a logical fallacy that distracts from the technical discussion, IMHO. Let's keep this on point.

The discussion was about swap, and you do not need to dump in swap space, you can dump in the filesystem, if you choose. The dump is configured with the Solaris utility dumpadm.

Obviously you do not have to dump in swap, so these arguments about core dumps and swap is a red-herring logical fallacy.

reborg said that you do need to allocate any swap to save cores. This is 100% correct.

This is indeed the point and I'm sorry to insist telling you are incorrect in thinking you can dump in the filesystem.

As I already wrote and as the documentation states you either dump in the swap area or in a dedicated partition. The latter doesn't help saving disk space, just the opposite. It helps speeding up the boot process which might make sense.

No, I am sorry to disagree on this point. The documentation for dumpadm(1M) and savecore(1M) both say that you specify the partition, but you can also specify the directory:

.

Excuse me for saying so, but /var/crash/hostname, for example, is a directory in a filesystem because directories are created in filesystems. (In other words, you have to make a filesystem before you can create a directory!)

And, as I mentioned before, this line of discussion has little to do with the issues reborg's point on swap space that you questioned, he posted:

and he was absolutely right.

You replied with your personal preference to dump in swap. That is your preference, not a requirement. While I respect your opinion, it is an opinion, not a requirement. You simply do not need to dump in swap.

If you take a look at this page:

dumpadm(1M) � configure operating system crash dump (man pages section 1M: System Administration Commands) - Sun Microsystems

You will see:

Example 1 Reconfiguring The Dump Device To A Dedicated Dump Device:

The following command reconfigures the dump device to a dedicated dump device:

That directory in the output of dumpadm is in the filesystem :smiley:

In the same man page as the post below:

I note that setting permissions on a directory (like 0700) can be done because it is a filesystem.