HELP - memory usage on Solaris : ps -efl and top

newbie_01 · August 30, 2010, 4:06pm

Hi all,

OS Version:
SunOS <hostname> 5.10 Generic_142900-13 sun4v sparc SUNW,Sun-Blade-T6340

I need some expert guidance on investigating memory usage on Solaris. I want to know whether am interpreting the output from ps -efl correctly and whether the command top is showing the right information or not. Details of the problem is as below:

top command reported free memory from this server is at 226MB out of 7968M reported. A bit worried that the server may run out of memory and start swapping. So far, for a week now, top reports no swapping so not that worried at this stage.
The top command memory usage per PID does not seems correct and is exceeding the 7968M available RAM, so we used ps -efl instead and the SZ column from running that ps command seems to match with the reported memory usage.
From the ps -efl output, the memory usage seems to be constant at 115M more or less, is there a server/kernel/database setting that sets minimum memory usage to start at 115M per process? This seem to be the case for oracle processes but can't find anything on the database settings that suggest this is getting set in the database ...
Any advise on how to troubleshoot this problem will be very much appreciated. Thanks in advance.

Several useful output information can be found below:

Output from "echo ::memstat | mdb -k" as below. I am hoping I can match up the output below with the ps -efl output somehow but not sure how to do that.

 
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     264913              2069   26%
ZFS File Data              437585              3418   44%
Anon                       191493              1496   19%
Exec and libs               24194               189    2%
Page cache                  20100               157    2%
Free (cachelist)             5420                42    1%
Free (freelist)             57134               446    6%
Total                     1000839              7819
Physical                   984896              7694

The server is used by a third-party application using JAVA and is also used as an Oracle database server.

ps -efl output from application connections as below. Based on the ps man pages, the SZ column is the memory usage in KB, is that correct, so that means from the output below the first process below is using 115005KB, roughly 115MB of memory, is that correct?

 
 F S      UID   PID  PPID   C PRI NI     ADDR     SZ    WCHAN    STIME TTY         TIME CMD
 0 S   oracle 14564     1   0  40 20        ? 115005        ?   Aug 27 ?           2:36 oracleECR (LOCAL=NO)
 0 S   oracle 14174     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14186     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14190     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14184     1   0  40 20        ? 114661        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14180     1   0  40 20        ? 114709        ?   Aug 27 ?           0:04 oracleECR (LOCAL=NO)
 0 S   oracle 14178     1   0  40 20        ? 114661        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14176     1   0  98 20        ? 114741        ?   Aug 27 ?           0:06 oracleECR (LOCAL=NO)
 0 S   oracle 14194     1   0  40 20        ? 115023        ?   Aug 27 ?           0:01 oracleECR (LOCAL=NO)
 0 S   oracle 14188     1   0  98 20        ? 115039        ?   Aug 27 ?           0:56 oracleECR (LOCAL=NO)
 0 S   oracle 14182     1   0  98 20        ? 114669        ?   Aug 27 ?           0:41 oracleECR (LOCAL=NO)
 0 S   oracle 14198     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14172     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14210     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14204     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14202     1   0  41 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14208     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14200     1   0  40 20        ? 114669        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14214     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14206     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14218     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14212     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14216     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14228     1   0  40 20        ? 114685        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14196     1   0  40 20        ? 114653        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)
 0 S   oracle 14192     1   0  40 20        ? 114613        ?   Aug 27 ?           0:00 oracleECR (LOCAL=NO)

ps -efl output for Oracle database processes as below. I assume I can't control the memory usage on these processes as they are spawned by Oracle.

 F S      UID   PID  PPID   C PRI NI     ADDR     SZ    WCHAN    STIME TTY         TIME CMD
 0 S   oracle  5932     1   0  42 20        ? 116208        ?   Aug 27 ?           0:02 ora_arc0_ECR
 0 S   oracle  5947     1   0  40 20        ? 114590        ?   Aug 27 ?           0:04 ora_qmnc_ECR
 0 S   oracle  4260     1   0  40 20        ? 116208        ?   Aug 27 ?           1:15 ora_lgwr_ECR
 0 S   oracle  4280     1   0  40 20        ? 114928        ?   Aug 27 ?           3:41 ora_ckpt_ECR
 0 S   oracle  4239     1   0  43 20        ? 115588        ?   Aug 27 ?           1:01 ora_dbw0_ECR
 0 S   oracle  4180     1   0  98 20        ? 114782        ?   Aug 27 ?           2:55 ora_pmon_ECR
 0 S   oracle  4203     1   0  40 20        ? 114590        ?   Aug 27 ?           0:18 ora_psp0_ECR
 0 S   oracle  4220     1   0  40 20        ? 114590        ?   Aug 27 ?           0:30 ora_mman_ECR
 0 S   oracle  4303     1   0  98 20        ? 114662        ?   Aug 27 ?           1:35 ora_smon_ECR
 0 S   oracle  4326     1   0  41 20        ? 114678        ?   Aug 27 ?           0:04 ora_reco_ECR
 0 S   oracle  4349     1   0  40 20        ? 114790        ?   Aug 27 ?           1:50 ora_mmon_ECR
 0 S   oracle  4372     1   0  40 20        ? 114646        ?   Aug 27 ?           8:12 ora_mmnl_ECR
 0 S     root  4383     1   0  41 20        ?  98815        ?   Aug 27 ?           0:17 ora_dism_ECR
 0 S   oracle  5934     1   0  40 20        ? 115950        ?   Aug 27 ?           0:10 ora_arc1_ECR
 0 S   oracle  6007     1   0  40 20        ? 114597        ?   Aug 27 ?           0:00 ora_q000_ECR
 0 S   oracle  6709     1   0  40 20        ? 114589        ?   Aug 27 ?           0:02 ora_q001_ECR
 0 S   oracle  5413     1   0  45 20        ? 114886        ? 21:31:01 ?           0:38 ora_cjq0_ECR

Excerpts from top ordered by SIZE. The output seems doubtful. From the output below, it shows each oracle PID consuming 800M of memory and I only have 7968M, how come? Should I discard these information and rely on the output from ps -efl then which seems to be the correct one?

last pid: 10464;  load avg:  0.20,  0.19,  0.18;       up 3+17:14:10                                                                05:39:48
153 processes: 151 sleeping, 1 zombie, 1 on cpu
CPU states: 98.9% idle,  0.4% user,  0.7% kernel,  0.0% iowait,  0.0% swap
Memory: 7968M phys mem, 216M free mem, 2048M swap, 2048M free swap
   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  4260 oracle    11  59    0  908M  881M sleep    1:15  0.01% oracle
  5932 oracle    11  57    0  908M  880M sleep    0:01  0.00% oracle
  5934 oracle     1  59    0  906M  880M sleep    0:09  0.00% oracle
  4239 oracle    55  56    0  903M  887M sleep    1:01  0.01% oracle
 14188 oracle    11   1    0  899M  886M sleep    0:55  0.00% oracle
 14194 oracle    11  59    0  899M  884M sleep    0:00  0.00% oracle
 14564 oracle     1  59    0  898M  887M sleep    2:36  0.00% oracle
  4280 oracle    11  59    0  898M  881M sleep    3:41  0.04% oracle
  5413 oracle     1  54    0  898M  887M sleep    0:38  0.11% oracle
  4349 oracle     1  59    0  897M  882M sleep    1:49  0.00% oracle
  4180 oracle     1   1    0  897M  880M sleep    2:55  0.04% oracle
 14176 oracle     1   1    0  896M  885M sleep    0:05  0.00% oracle
 14180 oracle     1  59    0  896M  884M sleep    0:03  0.00% oracle
 14228 oracle     1  59    0  896M  882M sleep    0:00  0.00% oracle
  4303 oracle     1   1    0  896M  881M sleep    1:35  0.00% oracle

ps -efl output from Non oracle processes as below:

 
 F S      UID   PID  PPID   C PRI NI     ADDR     SZ    WCHAN    STIME TTY         TIME CMD
 F S      UID   PID  PPID   C PRI NI     ADDR     SZ    WCHAN    STIME TTY         TIME CMD
 1 T     root     0     0   0   0 SY        ?      0            Aug 27 ?           0:25 sched
 0 S     root     1     0   0  98 20        ?    371        ?   Aug 27 ?           2:31 /sbin/init
 1 S     root     2     0   0   0 SY        ?      0        ?   Aug 27 ?           0:00 pageout
 1 S     root     3     0   0   0 SY        ?      0        ?   Aug 27 ?          86:33 fsflush
 0 S     root   532     1   0  70 20        ?    222        ?   Aug 27 ?           0:00 /usr/lib/efcode/sparcv9/efdaemon
 0 S     root     7     1   0  70 20        ?   1846        ?   Aug 27 ?           0:21 /lib/svc/bin/svc.startd
 0 S     root     9     1   0  70 20        ?   1593        ?   Aug 27 ?           0:54 /lib/svc/bin/svc.configd
 0 S     root   409     1   0  42 20        ?    325        ?   Aug 27 ?           0:00 /usr/lib/ldoms/drd
 0 S   daemon   980     1   0  70 20        ?    409        ?   Aug 27 ?           0:01 /usr/lib/nfs/statd
 0 S     root   449     1   0  70 20        ?    563        ?   Aug 27 ?           0:33 /usr/lib/picl/picld
 0 S     root   310     1   0  98 20        ?    776        ?   Aug 27 ?           0:08 devfsadmd
 0 S   daemon   414     1   0  70 20        ?    683        ?   Aug 27 ?           1:08 /usr/lib/crypto/kcfd
 0 S     root   419     1   0  70 20        ?   1281        ?   Aug 27 ?           0:00 /usr/lib/sysevent/syseventd
 0 S     root   424     1   0  44 20        ?   1165        ?   Aug 27 ?           8:40 /usr/sbin/nscd
 0 S     root  1013     1   0  46 20        ?    216        ?   Aug 27 ?           0:03 /usr/lib/utmpd
 0 S     root   469     1   0  98 20        ?    347        ?   Aug 27 ?           2:49 /usr/lib/inet/in.mpathd -a
 0 S     root   999     1   0  40 20        ?    963        ?   Aug 27 ?           1:13 /usr/lib/inet/inetd start
 0 S     root 10591     7   0  40 20        ?    366        ?   Aug 27 console     0:00 /usr/lib/saf/ttymon -g -d /dev/cons
 0 S   daemon   461     1   0  40 20        ?    962        ?   Aug 27 ?           0:39 /usr/lib/rcap/rcapd
 0 S     root 25488 25484   0  72 20        ?    671        ? 08:56:46 pts/4       0:00 bash --login
 0 S     root  1007   998   0  70 20        ?    366        ?   Aug 27 ?           0:00 /usr/lib/saf/ttymon
 0 S     root   512     1   0   0 RT        ?    398        ?   Aug 27 ?           0:36 /usr/lib/inet/xntpd
 0 S     root   515     1   0  70 20        ?    331        ?   Aug 27 ?           0:00 /lib/svc/method/iscsid
 0 S     root   554     1   0  41 20        ?    427        ?   Aug 27 ?           0:08 /usr/sbin/cron
 0 S     root   998     7   0  48 20        ?    306        ?   Aug 27 ?           0:00 /usr/lib/saf/sac -t 300
 0 S     root  1035     1   0  40 20        ?    577        ?   Aug 27 ?           0:05 /usr/lib/ssh/sshd
 0 S    smmsp  1062     1   0  40 20        ?   1096        ?   Aug 27 ?           0:02 /usr/lib/sendmail -Ac -q15m
 0 S     root  6746  6730   0  63 20        ?    453        ? 05:11:20 pts/5       0:00 -sh
 0 S     root  1040     1   0  40 20        ?    472        ?   Aug 27 ?           0:02 /usr/sbin/syslogd
 0 S     root  1058     1   0  57 20        ?    381        ?   Aug 27 ?           0:00 /usr/lib/snmp/snmpdx -y -c /etc/snm
 0 S   daemon   979     1   0  70 20        ?    648        ?   Aug 27 ?           0:24 /usr/lib/nfs/nfsmapid
 0 S     root  1061  1058   0  42 20        ?    454        ?   Aug 27 ?           5:11 mibiisa -r -p 32781
 0 S   daemon  1008     1   0  39  0        ?    350        ?   Aug 27 ?           0:00 /usr/lib/nfs/lockd
 0 S   daemon   973     1   0  42 20        ?    409        ?   Aug 27 ?           0:01 /usr/sbin/rpcbind
 0 S   daemon   975     1   0  39  0        ?    352        ?   Aug 27 ?           0:01 /usr/lib/nfs/nfs4cbd
 0 S     root  1060     1   0  49 20        ?   1290        ?   Aug 27 ?           0:06 /usr/sfw/sbin/snmpd
 0 O     root 10153  6750   0  40 20        ?    478          05:37:29 pts/5       0:00 ps -efl
 0 S     root  1064     1   0  40 20        ?   1080        ?   Aug 27 ?           0:14 /usr/lib/sendmail -bd -q15m
 0 S    ozone 25047 24984   0  40 20        ?    438        ? 07:06:57 pts/2       0:00 -bash
 0 S     root  1752  1308   0  40 20        ?    221        ?   Aug 27 ?           0:00 sh
 0 S     root  1261     1   0  46 20        ?    353        ?   Aug 27 ?           0:00 /opt/perf/bin/ttd
 0 S     root  1226     1   0  70 20        ?   1942        ?   Aug 27 ?           0:14 /usr/lib/fm/fmd/fmd
 0 S   daemon  1144     1   0  39  0        ?    357        ?   Aug 27 ?           0:00 /usr/lib/nfs/nfsd
 0 S     ixos 14542 14125   0  98 20        ?   3425        ?   Aug 27 ?           0:08 ixmonClnt -l Clnt_ds -k nul dspo ds
 0 S     root  1142     1   0  41 20        ?    505        ?   Aug 27 ?           0:01 /usr/lib/nfs/mountd
 0 S     root  4005     1   0  40 20        ?    710        ?   Aug 27 ?           0:05 /opt/OV/lbin/xpl/trc/ovtrcd
 0 S     root  1128     1   0  46 20        ?    435        ?   Aug 27 ?           0:00 /usr/sbin/vold -f /etc/vold.conf
 0 S     root  1754  1308   0  40 20        ?    217        ?   Aug 27 ?           0:00 sh
 0 S     root  1308     1   0  41 20        ?   2177        ?   Aug 27 ?          22:06 esd - init agent -dir /var/opt/SUNW
 0 S     root  4022  4004   0  40 20        ?   1551        ?   Aug 27 ?           0:02 /opt/OV/bin/ovbbccb -nodaemon
 0 S    ozone 10615 10612   0  98 25        ?    663        ?   Aug 27 ?           1:52 ./zabbix_agentd
 0 S    ozone 10613 10612   1  40 25        ?    662        ?   Aug 27 ?         197:05 ./zabbix_agentd
 0 S     root 25484 25450   0  40 20        ?    453        ? 08:56:46 pts/4       0:00 -sh
 0 S noaccess  2591     1   0  40 20        ?  15774        ?   Aug 27 ?           8:59 /usr/java/bin/java -server -Xmx128m
 0 S     root  2604   999   0  52 20        ?    481        ?   Aug 27 ?           0:00 /usr/sbin/rpc.metacld
 0 S     ixos 14158 14125   0  98 20        ?   5459        ?   Aug 27 ?           4:34 dsMain RC
 0 S    ozone 24984 24983   0  40 20        ?   1239        ? 07:06:28 ?           0:01 /usr/lib/ssh/sshd
 0 S     root  6063  4004   0  56 20        ?   2434        ?   Aug 27 ?           0:01 /opt/OV/lbin/eaagt/opcmsgi
 0 S     root 15420 15419   0  40 20        ?   1455        ?   Aug 29 ?           0:01 /usr/lib/ssh/sshd
 0 S     root 25449  1035   0  40 20        ?    679        ? 08:56:36 ?           0:00 /usr/lib/ssh/sshd
 0 S     root  3979     1   0  40 20        ?   1995        ?   Aug 27 ?           1:02 /opt/perf/bin/perfalarm
 0 S     root  4034  4004   0  40 20        ?   6087        ?   Aug 27 ?           1:50 /opt/OV/lbin/perf/coda
 0 S     root  3891     1   0   0 RT        ?    321        ?   Aug 27 ?           0:01 /opt/perf/bin/midaemon
 0 S     root  4004     1   0  45 20        ?   2234        ?   Aug 27 ?           2:36 /opt/OV/bin/ovcd
 0 S     ixos 14165 14125   0  98 20        ?  17329        ?   Aug 27 ?           3:30 /Ixos/archivesw/opt/java/bin/java -
 0 S    ozone 10617 10612   0  40 25        ?    639        ?   Aug 27 ?           0:16 ./zabbix_agentd
 0 S     root  4046  1308   0  41 20        ?    464        ?   Aug 27 ?           0:55 esd - shell perftool-shell.tcl
 0 S     ixos 14163 14125   0  87 20        ?   3832        ?   Aug 27 ?           0:14 volmig
 0 S     root  6730  6729   0  94 20        ?   1455        ? 05:11:16 ?           0:00 /usr/lib/ssh/sshd
 0 S     root  4097  1308   0  40 20        ?    217        ?   Aug 27 ?           0:02 sh
 0 S     root  4099  1308   0  40 20        ?    217        ?   Aug 27 ?           0:00 sh
 0 S     ixos 14125     1   0  40 20        ?    650        ?   Aug 27 ?           1:22 /Ixos/archivesw/bin/spawner
 0 S     root  6283  4004   0  41 20        ?   2379        ?   Aug 27 ?           9:58 /opt/OV/lbin/eaagt/opcle -std
 0 S     root  4383     1   0  41 20        ?  98815        ?   Aug 27 ?           0:17 ora_dism_ECR
 0 S     ixos 14152 14125   0  81 20        ?   3889        ?   Aug 27 ?           0:00 dtCC dsexec.log EXEC
 0 S     root  5521  1308   0  41 20        ?    217        ?   Aug 27 ?           0:02 sh
 0 S     root 25450 25449   0  40 20        ?   1455        ? 08:56:36 ?           0:00 /usr/lib/ssh/sshd
 0 S     root 14138 14129   0  40 20        ?  19042        ?   Aug 27 ?           2:51 /Ixos/archivesw/bin/jbd -sendSpawne
 0 S     ixos 14167 14125   0  44 20        ?    641        ?   Aug 27 ?           0:13 ixmonSvc -s
 0 S     ixos 14160 14125   0  90 20        ?   3675        ?   Aug 27 ?           0:02 bkupSrvr
 0 S     ixos 14159 14125   0  43 20        ?    863        ?   Aug 27 ?           0:14 dsSched
 0 S     ixos 14541 14125   0  40 20        ?    852        ?   Aug 27 ?           0:09 ixmonClnt -l Clnt_dp -k dpdir dpt d
 0 S     ixos 14151 14125   0  98 20        ?   3889        ?   Aug 27 ?           0:00 dtCC dscache2.log CACHE
 0 S     ixos 14129 14125   0  40 20        ?    233        ?   Aug 27 ?           0:00 /bin/ksh /Ixos/archivesw/bin/STORM.
 0 S     ixos 14127 14125   0  98 20        ?  15743        ?   Aug 27 ?           6:48 /Ixos/archivesw/opt/java/bin/java -
 0 S     ixos 14162 14125   0  98 20        ?   4872        ?   Aug 27 ?           2:17 admSrv
 0 S     ixos 14153 14125   0  76 20        ?   3889        ?   Aug 27 ?           0:00 dtCC dscopy.log COPY
 0 S     ixos 14155 14125   0  41 20        ?   4102        ?   Aug 27 ?           0:18 dsAuxSrvr
 0 S     ixos 14150 14125   0  80 20        ?   3889        ?   Aug 27 ?           0:00 dtCC dscache1.log CACHE
 0 S     ixos 14141 14125   0  40 20        ?    813        ?   Aug 27 ?           0:04 DP -spawner dsdp -cfd /Ixos/archive
 0 S     root  6064  4004   0  52 20        ?   1923        ?   Aug 27 ?           0:01 /opt/OV/lbin/conf/ovconfd
 0 S     ixos 14540 14125   0  44 20        ?    852        ?   Aug 27 ?           0:23 ixmonClnt -l Clnt_rpc -k rpc_port p
 0 S     ixos 14154 14125   0  88 20        ?   3889        ?   Aug 27 ?           0:00 dtCC dsremove.log REMOVE
 0 Z     root  6067  6064   0   0  -        -      0        -        - ?           0:00 <defunct>
 0 S     root  6287  4004   0  40 20        ?   2412        ?   Aug 27 ?           5:45 /opt/OV/lbin/eaagt/opcmsga
 0 S     ixos 14128 14125   0  40 20        ?    593        ?   Aug 27 ?           0:00 purgefiles access.[1-9][0-9][0-9][0
 0 S    ozone 10616 10612   0  98 25        ?    663        ?   Aug 27 ?           0:45 ./zabbix_agentd
 0 S     root  6298     1   0  98 39        ?    266        ?   Aug 27 ?           1:21 /usr/sbin/tictimed
 0 S    ozone 10614 10612   0  40 25        ?    663        ?   Aug 27 ?           3:25 ./zabbix_agentd
 0 S     root  6288  4004   0  47 20        ?   2849        ?   Aug 27 ?           2:41 /opt/OV/lbin/eaagt/opcmona
 0 S     root  6294  4004   0  40 20        ?   2407        ?   Aug 27 ?           1:08 /opt/OV/lbin/eaagt/opcacta
 0 S     root  8599 25298   0  40 20        ?    819        ? 05:25:35 pts/2       0:01 ssh vinhdt01
 0 S     root 14139 14138   0  40 20        ?    601        ?   Aug 27 ?           0:01 jsd /var/adm/ixos-archive/log 2 404
 0 S    ozone 10612     1   0  40 25        ?    639        ?   Aug 27 ?           0:00 ./zabbix_agentd
 0 S     root 15419  1035   0  40 20        ?    679        ?   Aug 29 ?           0:00 /usr/lib/ssh/sshd
 0 S     root 25298 25288   0  40 20        ?    438        ? 07:07:58 pts/2       0:00 bash --login
 0 S     root 15441 15420   0  40 20        ?    453        ?   Aug 29 pts/1       0:00 -sh
 0 S     root  6750  6746   0  40 20        ?    671        ? 05:11:20 pts/5       0:00 bash --login
 0 S     ixos 14157 14125   0  98 20        ?   5365        ?   Aug 27 ?           4:45 dsMain WC
 0 S     root 25288 25047   0  40 20        ?    221        ? 07:07:56 pts/2       0:00 -sh
 0 S     root 15445 15441   0  61 20        ?    671        ?   Aug 29 pts/1       0:01 bash --login
 0 S     root 24983  1035   0  40 20        ?    843        ? 07:06:28 ?           0:02 /usr/lib/ssh/sshd
 0 S     root  6729  1035   0  98 20        ?    679        ? 05:11:16 ?           0:00 /usr/lib/ssh/sshd

jlliagre · August 30, 2010, 4:33pm

"top" isn't part of the OS. It is known to report confusing data as it doesn't use always the same terminology as Solaris does. Better to avoid it and use prstat, vmstat, "swap -s" and the likes.

There is hopefully no minimum memory usage kernel tuning per process. I would guess this is simply due to shared memory accounted to each process.

What problem do you want to troubleshoot ? You tell there is no swapping reported so IMHO nothing to fix. You have plenty of available memory (most of the ZFS file data is available to applications should they need it) and anyway, unused memory is wasted memory.

newbie_01 · September 20, 2010, 1:19am

Hi,

I totally agreed with what you noted so I will stick to the known reliable generic commands of vmstat, prstat etc. I got some more questions below, if you don't mind ... o

Again, totally agreed, no swapping = no problem = nothing to fix. And you are right, unused memory is wasted memory, it's like why worry about 90% CPU used as long as it is a valid not for aethestic reason, i.e. looks good/bad on the graph
In saying that, what am mainly wanting to achieve I suppose is whether the reported memory usage from ps -efl column is correct or not. If I sum up the values under the SZ column and the total is less than the total RAM on the server, does that mean, swapping is not happening. Will there be a case where the sum of the SZ column is greater than the total RAM on the server in which case, that should mean swapping is happening? BTW, is it possible to get total memory usage using the ps command ... ... am currently doing this via an awk script.
:oYou said "You have plenty of available memory (most of the ZFS file data is available to applications should they need it)", how are you able to tell that it is available or unavailable? Am presuming you are getting this info from the "echo ::memstat | mdb -k". Doesn't the output from that mean used/allocated memory by "ZFS File Data" rather than whether it is available/unnavailable memory or is that a wrong assumption? Can you please explain further?
The "no swapping" conclusion is from the top command so without using top, my vmstat output is as below, po is zero (0) so that confirms that swapping is not happening.

vmstat output:

 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr vc vc vc --   in   sy   cs us sy id
 0 0 0 970584 531248 244 768 145 0 0  0  0  0  0  2  0 1181 2719 1424  2  2 96
 0 0 0 967432 526872 979 1311 13 0 0  0  0 18  0  2  0 1025 5305 1159  7  7 86
 0 0 0 968808 529360 125 470 16 0  0  0  0  0  0  1  0  772 2869 1076  1  3 96
 0 0 0 969936 530528 32 163  2  0  0  0  0  0  0 23  0  803  820  722  0  1 98
 0 0 0 968120 530472 32 163 24  0  0  0  0  0  0  2  0  750  923  628  0  1 98
 0 0 0 970072 530520 47 217  5  0  0  0  0  0  0  4  0  900 1585  914  1  1 97
 0 0 0 970072 530480 32 163 10  0  0  0  0  4  0  2  0  817  897  746  1  1 98
 0 0 0 970072 530424 45 245  2  0  0  0  0  4  0  3  0  786 1183  696  1  1 98
 0 0 0 970072 530392 33 163 16  0  0  0  0  0  0 31  0  987 1050 1001  1  1 98
 0 0 0 969968 530104 29 127  0  0  0  0  0  0  0  3  0  660  788  548  1  1 98
 0 0 0 969944 530072 94 259  0  0  0  0  0  0  0  1  0  685 1983  565  1  2 97
 0 0 0 969504 529392 565 1207 3 0  0  0  0  0  0  1  0  857 5262 1115  5  3 92
 0 0 0 969624 529416 268 555 3  0  0  0  0  3  0  2  0  934 2734 1063  2  2 96
 0 0 0 965576 524256 1635 3352 37 0 0 0  0  0  0  2  0 1312 13458 2363 15 12 73

jlliagre · September 20, 2010, 3:42am

Likely, but not necessarily.

Possibly but not necessarily.

"ps" doesn't do aggregation. You might want to use "prstat -Z" to get that kind of statistics. Beware that kernel RAM usage isn't reported by ps or prstat as the kernel isn't a process, at least not a regular one.

Because that's by design. ZFS is the only file system I know that reports its cache as used memory. Other file systems report the equivalent as free memory. This is confusing a lot of ZFS users.

A large part of it is, as I just wrote, available to process for allocations so is really free memory.

Better to watch vmstat "sr" column, as long as it stays equals or close to zero, there is no worry to have about swapping.