Solaris Mem Consumption

We have Sun OS running on spark :

SunOS ciniwnpr67 5.10 Generic_118833-24 sun4u sparc SUNW,Sun-Fire-V440

Having Physical RAM :

Sol10box # prtconf | grep Mem
Memory size: 8192 Megabytes

My Top Output is :

130 processes: 129 sleeping, 1 on cpu
CPU states: 98.8% idle, 0.2% user, 1.0% kernel, 0.0% iowait, 0.0% swap
Memory: 8192M real, 343M free, 2282M swap in use, 2195M swap free

Which shows about 7 gigs of RAM already consumed.

My sar -k shows :

SunOS ciniwnpr67 5.10 Generic_118833-24 sun4u 08/07/2008

07:29:53 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail
07:29:54 4184032128 4044621477 0 901931008 621211704 1 1723777024 0

And if i under stood it right then i have around 4044621477 + 621211704 + 1723777024 Bytes of memory or 4.5 gigs used by kernel itself ??

If it is true then it i think some thing is wrong with my solaris installtion Any suggestions pls ?

whats the output of prtstat -a ?

PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  7256 root       62M 7272K cpu1     0    0 336:11:33  24% coda/3
 15251 csvn       74M   45M sleep   59    0   0:43:01  10% httpd/1
 26773 root       13M   10M sleep   60    0   0:02:53 5.3% svn/1
  1603 root      268M   76M sleep   29   10   3:34:16 0.8% java/42
  1094 root     7800K  280K sleep   59    0  19:39:19 0.1% rmserver/6
 27463 root     9104K 6072K sleep   59    0   0:00:00 0.1% sshd/1
 11966 iwui       13M 1232K sleep   59    0   0:10:03 0.1% LLAWP/4
 27418 apache0    62M   25M sleep   29   10   0:00:00 0.1% java/11
 27501 root     4944K 4600K cpu0    49    0   0:00:00 0.0% prstat/1
 11818 iwts      102M   94M sleep   59    0   1:08:33 0.0% iwserver.sol/57
 11988 iwui      385M  225M sleep   59    0   0:30:15 0.0% java/52
 27496 root     3000K 2432K sleep   49    0   0:00:00 0.0% bash/1
 24833 rajwinde 8848K 3608K sleep   59    0   0:00:00 0.0% sshd/1
   950 root     4480K 1816K sleep   59    0   2:58:50 0.0% cstd.agt/14
 17892 iwui       62M   15M sleep   29   10   0:09:04 0.0% java/11
 27479 rajwinde 8848K 3680K sleep   59    0   0:00:00 0.0% sshd/1
 27492 rajwinde 4152K 2312K sleep   59    0   0:00:00 0.0% pbrun/1
  2104 tomcat    270M   85M sleep   59    0   0:17:32 0.0% java/38
     9 root       10M 4432K sleep   59    0   0:06:35 0.0% svc.configd/13
 24880 root     4168K 2056K sleep   59    0   0:00:00 0.0% pblocald/1
 24878 rajwinde 4152K 2232K sleep   59    0   0:00:00 0.0% pbrun/1
  1082 root     6544K  200K sleep   59    0   2:05:03 0.0% mstragent/1
  1145 root     8272K 1784K sleep   59    0   2:17:38 0.0% mstragent/6
 27494 root     4168K 2080K sleep   59    0   0:00:00 0.0% pblocald/1
   783 root       77M 6752K sleep   59    0   2:25:43 0.0% vxsvc/42
 27486 root     1936K 1504K sleep   59    0   0:00:00 0.0% rquotad/1
 27481 rajwinde 1304K 1184K sleep   59    0   0:00:00 0.0% sh/1
 27410 apache0  7360K 6104K sleep   59    0   0:00:00 0.0% Deploy_pacData./1
  2111 root       44M   10M sleep   59    0   4:21:50 0.0% msragent/10
   145 daemon   5432K 1752K sleep   59    0   0:00:46 0.0% kcfd/4
   580 root     5344K 1448K sleep   59    0   0:05:02 0.0% inetd/4
  7252 root       21M   11M sleep   59    0   1:09:43 0.0% opcmona/11
   869 root       29M 4128K sleep   59    0   1:30:08 0.0% python/9
 NPROC USERNAME  SIZE   RSS MEMORY      TIME  CPU
    86 root     1002M  259M   3.1% 385:22:12  31%
     8 csvn      248M   58M   0.7%   1:16:32  10%
     9 iwui      569M  263M   3.3%   0:53:50 0.1%
     9 rajwinde   42M   21M   0.2%   0:00:00 0.1%
    14 apache0   139M   51M   0.6%   0:00:07 0.1%
     4 iwts      117M   97M   1.2%   1:10:16 0.0%
     1 tomcat    270M   85M   1.1%   0:17:32 0.0%
     7 daemon     32M 4880K   0.1%   0:26:05 0.0%
     3 jayesh     14M 3800K   0.0%   0:00:00 0.0%
     1 smmsp    7472K  712K   0.0%   0:00:05 0.0%
Total: 142 processes, 751 lwps, load averages: 1.77, 1.23, 0.84

Your system looks OK to me.
Pls update your kernel patches (esp Recommended cluster) to the latest of Solaris 10. There's much bug fixes to 118833-24 . No worries..
:b:

:pAnd not to forget, as a rule of thumb, swap device should be configured double the size than of your physical memory. (16384)
V440 is a stable server. Yours should keep going.

well here is the issue ..:

As per Top i am using 7 gigs of RAm.. now the moment i run some thing else then the current applications it page outs increases significantly from almot "0" to 4ooo +

That is causing the new processs to take more time ..

check this out
How to Check Kernel Memory Allocation (sar -k) (System Administration Guide: Advanced Administration) - Sun Microsystems
do this:
# prtmem
#netstat -kp | grep pp_kernel
#mdb -k

here is the thing :

# prtmem - is a part of memtool and that is not recomended to run on prod i guess

#netstat -kp | grep pp_kernel
For me there is no option as - k in netstat on solaris 10

#mdb -k

I really dont have any idea about this dubugger but i will chk it out ..

Tell me this thoguh what sar -k shows in my case .. didnt that meant that kernel is using so much memory ?

That is a very old rule of thumb which I no longer agree with. Memory is so cheap and plentiful these days that you should hardly ever require swap, if you do it is a sign of problems. I generally don't ever configure more than 4GB of swap, even on a 64+GB system.

In fact, if your physical memory is alot, of course Im not asking you to give 128 GB for your swap. But you will t least need 16Gb of swap for a 64 0r 128 configuration minimally.

This is the architecture of Solaris. You will need additional swap for your paging and the space for your coredump(which might be Huge in size). You won't want your system to crash further after a system panic, would you?:wink:

incredible - of course your dump could be rather large but i run m9000's and have never seen a core file greater then a few gigs tops. we are talking about 1-4 TBs of memory and 64 sparc64 VI (128 cores) processors.

For your info M series is a collaboration of Fujitsu with SUN and their architectures do differ.. Though you may find M9000 is working like a 6900 system.. :stuck_out_tongue:
If its gonna be a "SUN" mid-range or highend servers, what I say still makes sense.

Yes, I agree with this. The old 2 pounds of swap for every pound of memory was back in the old days when memory was expensive and it was cheaper to use disk swap.

If you have 7 GB of RAM, something is really wrong if you need 14GB of swap.

4GB of swap is more than enough, good post Annihilannic. Thanks.

There is a lot of infomation in this thread which is not valid. Almost everything Incredible has stated in this thread is incorrect in terms both of system configurations and reasons for choosing swap size.

A few points worth note.

  1. You don't need to allocate any swap space to deal with core dumps, and have not since Sorlais 8.
  2. Solaris will never go into a panic-reboot cycle as a result of not having savecore space. It will simply not save a core dump if it has no space.
  3. Twice memory as swap is no longer a good choice unless you really can't afford to upgrade.
  4. If you have a lot of pagout you do not have enough memory, it's as simple as that.
  5. You do not need minimum 16GB swap for 64 or 128GB of memory, but you may need to have more swap if you have applications using ISM (Intimate shared memory) or DISM (Dynamic Intimate Shared Memory) such as Sybase or Oracle databases.
  6. There is no reason to treat an M-Series differently from any other (SPARC) Solaris box.

In summary you can get by to some extent if you don't have enough memory by adding swap, but it will hurt performance. Ideally you should have enough memory to run all your applications in memory, and the general rule of thumb nowadays is about 30% of memory for swap but there are more detailed reccomendations in the Solaris documentation.

Jim Laurent at Sun wrote a blog on this topic about a year ago, which you could look up.

EDIT: Found the blog:Solaris FAQ: Myths and facts about Solaris swap space : Jim Laurent's Weblog

If you want to debate the accuracy of anything in this thread, do so in this thread.

If posts are not related to the original topic they will be deleted.

Are you using ZFS on this machine?

Glad you asked about ZFS, I was about to do the same.

The OP has 8 GB of RAM, 1.5 GB is used by processes while 6.5 GB is used by the kernel. This per se isn't a problem. Unused RAM is wasted RAM anyway.

One major change ZFS introduced compared to UFS is the page cache is not used to cache file content. Kernel memory is used instead . This dramatically increases the kernel RAM usage metric but has no real consequence as this cache memory is still free memory from a system's viewpoint.

On the other hand, if the OP isn't using ZFS, then there is a problem to investigate further. It is particularly a problem because the kernel seems to be already larger than the swap space and might grow even bigger before a potential panic. Assuming the dumpadm intricacies are unknown to the system administrator, a crash dump might be truncated and then be partially or, worst case scenario, wholly unusable.

The reason why I was picky about that point in previous postings.

Yes, indeed that was the reason for asking. The behavior you describe is was is expected from the ARC cache, however it is worth noting there have been a number of bugs related to the cache memory not being freed quickly enough, so it doesn't always work the way it should resulting in unnecessary paging. I have actually run into this issue, but I believe I have only seen it on x86 machines. It is possible, and advisable, to limit the ZFS cache for this reason if there is a possibility that it will interfere with other operations.

BTW,

I was reviewing the docs on the ZFS filesystem and dumping and found this example:

Using a ZFS Volume as a Swap or Dump Device (Solaris ZFS Administration Guide) - Sun Microsystems

# dumpadm
      
Dump content: kernel pages
Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/t2000-bdr-02
Savecore enabled: yes

Beware that this is beta documentation showing a feature AFAIK not yet released in Solaris 10.

8.ZFS Advanced Topics (Solaris ZFS Administration Guide) - Sun Microsystems states:

While Zvol can be used, a plain file on ZFS as a dump device is not supported even with Solaris Express, and there is no plan to implement it.