VM becomes slow every 3-4 days

Linux and Unix-Like Red Hat
1

Hi,
I have RHEL 6.1 on VM, where Big data is installed. Every 3-4 days, its web interface become very slow and I had to reboot this VM. I was checking with "top" and that shows that free memory is less. Can this be a reason for slowness ? No big load on CPU.

[root@server-01 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          7873       7753        120          0          2        276
-/+ buffers/cache:       7474        399
Swap:        10063       4403       5660
[root@server-01 ~]# vmstat 1 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0 4509316 120036   2644 283144   24   15    34    51   24    6  2  1 96  1  0
 4  0 4509296 119912   2644 283304   32    0    32     0 1686 6463  2  0 98  0  0
[root@server-01 ~]#
[root@server-01 ~]# ps aux | awk '{print $2, $4, $11}' | sort -k2r | head -n 10
PID %MEM COMMAND
1913 3.3 /usr/java/jdk1.6.0_31/bin/java
4232 3.3 /usr/java/jdk1.6.0_31/bin/java
2333 2.0 java
3397 1.8 /usr/java/jdk1.6.0_31/bin/java
1784 1.6 postgres:
4230 1.5 /usr/java/jdk1.6.0_31/bin/java
22477 1.3 /usr/java/jdk1.6.0_31/bin/java
3058 1.3 /usr/java/jdk1.6.0_31/bin/java
3995 1.3 /usr/java/jdk1.6.0_31/bin/java
[root@server-01 ~]# top
top - 11:33:32 up 3 days, 23:49,  2 users,  load average: 1.59, 0.51, 0.59
Tasks: 218 total,   1 running, 217 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.5%us,  1.0%sy,  0.0%ni, 24.6%id, 69.7%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   8062732k total,  7956004k used,   106728k free,     1860k buffers
Swap: 10305528k total,  4508972k used,  5796556k free,   259976k cached

Not sure, if I need to limit heap size or edit sysctl file. Please suggest.

2

Any suggestions on this issue ?

3

Not unexpectedly, Java appears to be the culprit. Has this only started recently?

4

You provided VERY little information when you asked this question. So, it isn't surprising that there weren't any responses. Bumping up the post with no new information doesn't help.

You haven't shown us anything that indicates that anything is running slowly; only that the amount of memory used by your top 10 memory consumers is less than 20% of your system's memory.

5

Don, only thing I can see that its web interface is very slow. Free memory left is low. From OS side, what else should I check ?
Yes this is new vm which was spinned off one month back.

6

What processes are eating up CPU?

How much net traffic is there? Are packets being dropped or retried?

You have one active task and 217 sleeping tasks? What are they waiting for?

Do you have zombies filling up your process table?

If this is a Big data server, how do the disk queues look? Are you getting I/O errors?

Is anything else going on every 3-4 days? Does it happen at the same time-of-day? Do you immediately reboot when it starts to get slow? Once it starts getting slow, does it stay consistently slow? Does it get increasingly slower?

Are other VMs on this machine seeing any increased activity or slower response times too?

7

Free memory looks low in the free -m output. If memory use is steadily increasing over those 3-4 days then you would need to look at that Java application. Is the heap size controlled, or is it allowed to just keep growing? Is VMStat showing high paging / swapping activity (there is some paging out in the output you posted)? If so, I would give the VM more memory.