Memory Issue

Linux and Unix-Like Red Hat
1

I could not find what is consuming the memory, generated DSET reports and NO hardware wise memory issue. 64 GB RAM on a server yet all I could see is a very limited memory available. I am not sure if I am reading this correct or not. I have used free -t -m and cat /proc/meminfo (results below)

  • No users on this server yet
    -One instance of an application running its owner ID is app_1
top output
top - 13:08:39 up 100 days, 14:44,  1 user,  load average: 0.33, 0.30, 0.23
Tasks: 695 total,   2 running, 692 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.4%us,  0.3%sy,  0.0%ni, 99.0%id,  0.4%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  65923324k total, 62353432k used,  3569892k free,  1057088k buffers
Swap: 33554424k total,     7652k used, 33546772k free, 49536116k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27025 app_1      20   0 14.0g 4.3g  14m S  3.6  6.9 107:44.25 java
30791 app_1      20   0 7200m 305m  16m S  1.6  0.5  31:18.50 java
 2329 app_1       20   0 15528 1744  952 R  1.3  0.0   0:00.24 top
26885 app_1       20   0 6669m 505m  13m S  1.3  0.8  37:45.53 java
28084 app_1       20   0 10.7g 1.2g  11m S  1.3  1.9  23:36.21 java
 3187 root      20   0  246m  25m 6764 S  0.7  0.0 423:33.62 kdm_greet
14876 app_1       20   0 19.9g 351m  10m S  0.3  0.5   5:05.44 java
25519 app_1       20   0  626m  11m 3140 S  0.3  0.0   2:08.21 python
25552 app_1       20   0 1003m  13m 3360 S  0.3  0.0   7:34.87 python
27089 app_1       20   0  419m  29m  25m S  0.3  0.0   3:35.61 postgres
27984 app_1       20   0 2491m 203m  14m S  0.3  0.3   1:56.11 java
    1 root      20   0 19356 1380 1144 S  0.0  0.0   0:54.37 init

OS = 2.6.32-358.6.2.el6.x86_64 #1 SMP  x86_64 x86_64 x86_64 GNU/Linux
cmd used:
free -t -m
             total       used       free     shared    buffers     cached
Mem:         64378      60887       3490          0       1032      48374
-/+ buffers/cache:      11480      52897
Swap:        32767          7      32760
Total:       97146      60894      36251

$ cat /proc/meminfo
MemTotal:       65923324 kB
MemFree:         3573696 kB
Buffers:         1056912 kB
Cached:         49535124 kB
SwapCached:          696 kB
Active:         30934280 kB
Inactive:       27772724 kB
Active(anon):    8099696 kB
Inactive(anon):   177692 kB
Active(file):   22834584 kB
Inactive(file): 27595032 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      33554424 kB
SwapFree:       33546772 kB
Dirty:               492 kB
Writeback:             0 kB
AnonPages:       8114300 kB
Mapped:           225212 kB
Shmem:            162420 kB
Slab:            3065196 kB
SReclaimable:    2793860 kB
SUnreclaim:       271336 kB
KernelStack:       17736 kB
PageTables:        45400 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    66516084 kB
Committed_AS:   16511580 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      387288 kB
VmallocChunk:   34359312380 kB
HardwareCorrupted:     0 kB
AnonHugePages:   7522304 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        5056 kB
DirectMap2M:     2045952 kB
DirectMap1G:    65011712 kB

Any insight is deeply appreciated.

2

You have 48 gigabytes of cache, which counts as free! You are not out of memory! Calm down and step back. You're fine. :slight_smile:

Linux and most other time-sharing OSes use free memory as disk cache, since it's got nothing better to do with it. It gives it up as freely as "free" at need.

You would not believe the lengths some people go to to 'flush the cache', up to and including rebooting the server daily, to solve something they only thought was a problem.

3

Phew!!!! Much thanks sir. I do find some PIDS under app_1 ownership that are showing VSZ of 19 GB so I panicked. I did

ps -aux|less

for the VSZ and RSS listing.

Stepping back :slight_smile: & Thank you for your time and assurance.

4

RSS is "resident set size", the amount of actual RAM a program is using. It's the one to worry about, usually. VSS is the virtual size, which is often far bigger than the amount of memory actually used. Much memory is shared between programs, too, for libraries and such, which inflates the size of VSS even more -- just blindly adding up VSS sizes to a total may add the same memory many times.

The shared memory problem is also true for RSS in fact, which can make programs which have large amounts of data shared, like oracle, difficult to pin down exactly how much they're using at a time.