Swap memory usage keeps increasing.

tom8254 · May 24, 2017, 9:02am

Hi

the version is RedHat 6.2 (Oracle DB server)

I don't know why swap memory usage keeps increasing...

I used to check swap memory usage Free -m and another way.

             total       used       free     shared    buffers     cached
Mem:         32183      31861        322          0        409      28413
-/+ buffers/cache:       3038      29145
Swap:        10236       5325       4911

and used below command to check each PID's swap memory usage

[root@ ~]# for proc in /proc/*; do cat $proc/smaps 2>/dev/null | awk '/Swap/{swap+=$2}END{print swap "\t'`readlink $proc/exe`'"}'; done | sort -n | awk '{total+=$1}/[0-9]/;END{print total "\tTotal"}'
0       /bin/bash
0       /bin/bash
0       /bin/bash
0       /bin/bash
0       /bin/bash
0       /bin/dbus-daemon
0       /bin/gawk
0       /bin/sort
0       /bin/su
0       /opt/hp/hp-health/bin/hpasmlited
0       /opt/hp/hp-health/bin/hp-asrd
0       /opt/hp/hp-snmp-agents/nic/bin/cmanicd
0       /opt/hp/hp-snmp-agents/server/bin/cmahealthd
0       /opt/hp/hp-snmp-agents/server/bin/cmahostd
0       /opt/hp/hp-snmp-agents/server/bin/cmapeerd
0       /opt/hp/hp-snmp-agents/server/bin/cmaperfd
0       /opt/hp/hp-snmp-agents/server/bin/cmasm2d
0       /opt/hp/hp-snmp-agents/server/bin/cmasm2d
0       /opt/hp/hp-snmp-agents/server/bin/cmastdeqd
0       /opt/hp/hp-snmp-agents/server/bin/cmathreshd
0       /opt/hp/hp-snmp-agents/storage/bin/cmaeventd
0       /opt/hp/hp-snmp-agents/storage/bin/cmafcad
0       /opt/hp/hp-snmp-agents/storage/bin/cmaidad
0       /opt/hp/hp-snmp-agents/storage/bin/cmaided
0       /opt/hp/hp-snmp-agents/storage/bin/cmasasd
0       /opt/hp/hp-snmp-agents/storage/bin/cmascsid
0       /opt/VRTSralus/bin/beremote
0       /oracle/app/11.2/grid/bin/crsd.bin
0       /oracle/app/11.2/grid/bin/cssdagent
0       /oracle/app/11.2/grid/bin/cssdmonitor
0       /oracle/app/11.2/grid/bin/evmd.bin
0       /oracle/app/11.2/grid/bin/evmlogger.bin
0       /oracle/app/11.2/grid/bin/gipcd.bin
0       /oracle/app/11.2/grid/bin/gpnpd.bin
0       /oracle/app/11.2/grid/bin/mdnsd.bin
0       /oracle/app/11.2/grid/bin/ocssd.bin
0       /oracle/app/11.2/grid/bin/octssd.bin
0       /oracle/app/11.2/grid/bin/ohasd.bin
0       /oracle/app/11.2/grid/bin/ologgerd
0       /oracle/app/11.2/grid/bin/oraagent.bin
0       /oracle/app/11.2/grid/bin/oraagent.bin
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/oracle
0       /oracle/app/11.2/grid/bin/orarootagent.bin
0       /oracle/app/11.2/grid/bin/orarootagent.bin
0       /oracle/app/11.2/grid/bin/osysmond.bin
0       /oracle/app/11.2/grid/bin/tnslsnr
0       /oracle/app/11.2/grid/opmn/bin/ons
0       /oracle/app/11.2/grid/opmn/bin/ons
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /oracle/app/oracle/product/11.2/bin/oracle
0       /root/nmon/bin/nmon_x86_64_rhel5
0       /sbin/audispd
0       /sbin/auditd
0       /sbin/init
0       /sbin/klogd
0       /sbin/mingetty
0       /sbin/mingetty
0       /sbin/mingetty
0       /sbin/mingetty
0       /sbin/mingetty
0       /sbin/mingetty
0       /sbin/multipathd
0       /sbin/portmap
0       /sbin/rpc.statd
0       /sbin/syslogd
0       /sbin/udevd
0       /skcc/ACEMAgent/3.0/bin/ACEMAgent.exe
0       /skcc/ACEMAgent/3.0/bin/INWatchDog.exe
0       /usr/bin/python
0       /usr/bin/readlink
0       /usr/bin/xfs
0       /usr/bin/Xorg
0       /usr/lib/Acronis/Agent/acronisagent
0       /usr/lib/Acronis/BackupAndRecovery/arsm
0       /usr/lib/Acronis/BackupAndRecovery/mms
0       /usr/libexec/gdmgreeter
0       /usr/libexec/gdm-rh-security-token-helper
0       /usr/libexec/hald-addon-acpi
0       /usr/libexec/hald-addon-keyboard
0       /usr/libexec/hald-addon-keyboard
0       /usr/libexec/hald-addon-keyboard
0       /usr/libexec/hald-addon-storage
0       /usr/libexec/hald-runner
0       /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java
0       /usr/sbin/atd
0       /usr/sbin/cimserver
0       /usr/sbin/crond
0       /usr/sbin/gdm-binary
0       /usr/sbin/gdm-binary
0       /usr/sbin/gpm
0       /usr/sbin/hald
0       /usr/sbin/hpiod
0       /usr/sbin/irqbalance
0       /usr/sbin/ntpd
0       /usr/sbin/pcscd
0       /usr/sbin/rpc.idmapd
0       /usr/sbin/sshd
0       /usr/sbin/sshd
0       /usr/sbin/sshd
0       /usr/sbin/sshd
0       /usr/sbin/sshd
0       /usr/sbin/uuidd
0       /usr/sbin/xinetd
0       Total

but....the result value is 0.....

I don't know why there is no swap memory usage but there is existing swap memory usage in free -m command's result.

Can you help me why the swap memory usage keeps increasing?

I attach another command result

[root@~]# vmstat 3 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 5453296 294920 419980 29099820    0    0   240    85    0    1  1  1 99  0  0
 0  0 5453296 294548 419992 29099828    0    0    32   171 1207 5943  0  0 99  0  0
 0  0 5453296 292984 420004 29099824    0    0    27   162 1200 5613  0  0 99  0  0
 0  0 5453296 293480 420016 29099828    0    0    48   554 1145 5199  0  0 99  0  0
 0  0 5453296 294224 420024 29099840    0    0    28    50 1129 6146  0  0 99  0  0
 1  0 5453296 295864 420040 29099840    0    0    33  1172 1154 5213  0  0 99  0  0
 0  0 5453296 295144 420068 29099832    0    0    53   107 1269 5804  0  0 99  0  0
 0  0 5453296 299752 420092 29099940    0    0   112   118 1137 5295  0  0 99  0  0
 0  0 5453296 299196 420092 29099944    0    0    53    74 1121 5331  1  0 99  0  0
 0  0 5453296 299988 420104 29099952    0    0    43   166 1111 5427  0  0 99  0  0

top - 21:02:39 up 71 days,  4:41,  2 users,  load average: 1.28, 1.19, 1.11
Tasks: 394 total,   1 running, 393 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  0.3%sy,  0.0%ni, 98.8%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND
13547 acem      18   0  522m 112m 1684 S  2.0  0.3   1005:09 410m ACEMAgent.exe
11918 root      15   0  352m  35m  14m S  1.0  0.1 491:31.07 317m orarootagent.bi
11977 oracle    RT   0  382m 110m  53m S  1.0  0.3 540:01.24 271m ocssd.bin
17219 root      18   0 12216 1264  600 S  1.0  0.0   0:32.60  10m nmon_x86_64_rhe
13262 root      15   0 1707m  25m  13m S  0.7  0.1 472:50.90 1.6g orarootagent.bi
18120 oracle    15   0 12.2g  76m  50m S  0.7  0.2 406:27.19  12g oracle
 5861 root      RT   0  150m 4280 2816 S  0.3  0.0  85:00.28 145m multipathd
11394 root      15   0  381m  51m  21m S  0.3  0.2 110:03.00 329m ohasd.bin
11832 oracle    15   0  371m  36m  15m S  0.3  0.1 327:39.28 334m oraagent.bin
11934 root      RT   0  246m  87m  55m S  0.3  0.3 378:02.72 159m osysmond.bin
13597 oracle    15   0 12.2g  47m  42m S  0.3  0.1 163:53.72  12g oracle
18726 oracle    15   0 12.2g  47m  42m S  0.3  0.1 202:53.48  12g oracle
18728 oracle    15   0 12.2g  58m  53m S  0.3  0.2 187:58.09  12g oracle
25375 oracle    15   0 12.2g  46m  42m S  0.3  0.1 197:19.75  12g oracle
25377 oracle    15   0 12.2g  48m  43m S  0.3  0.2 197:12.23  12g oracle
27058 oracle    15   0 12.2g  72m  67m S  0.3  0.2 182:48.25  12g oracle
28400 root      15   0 13000 1376  848 R  0.3  0.0   0:00.04  11m top

[root@ ~]# sar -r | grep Average
Average:       299196  32657104     99.09    379724  29115012   5151813   5330559     50.85    219135

Peasant · May 24, 2017, 12:03pm

From what you have posted, during the time you executed those commands there was no swapping.
But, your system did swap some period of time, hence the metrics.

Considering the cached column in your output, your system has cached 29 GB of data in memory.

For your output i can see you are running databases, is this Oracle ASM or filesystems in question ?

If you are running Oracle ASM, i would recommend tuning the kernel file system caching parameters and some others if required.

Here is a nice link :
https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6\_1.2_1.pdf

The swappiness parameters is most important to notice in the document and apply to your environment.

If filesystems are in question, it would be required to pinpoint the specific time of memory starvation using metrics from cron and reports from users.
The kernel tuning if you are using filesystems should be done differently, since your database / system benefits greatly from memory for read io.
Read doesn't hit disks so often, so they have more time to service write request and repopulating cache, making system faster.

Corona688 · May 24, 2017, 1:39pm

I suspect that culprit and owner are both Oracle. What else but Oracle would fight to keep 5 gigabytes of idle memory just sitting around? Oracle keeps its own caches, not relying on the hosts, and they could end up in swap if they're sitting idle when Oracle outgrows system memory. This is why it's so important to tune Oracle to your system.

Swapped cache is an amusing concept.

Corona688 · May 24, 2017, 1:45pm

Only sort of. The kernel will readily give up cache when needed. Large amounts of cache show a system that's not currently under memory pressure, but could be doing a fair amount of I/O. Large amounts of swap show a system that was under memory pressure at some point in the past but may not be now. Stuff that ends up in swap tends to stay, since idle memory gets swapped, not busy memory; it'll leave swap when asked for and no sooner.