Linux machine performance issue.

pinga123 · August 6, 2010, 1:03am

One of our database server is suddenly became very slow and i have no clue what to do .Please help. I m sharing the performance inforamtion regarding cpu,harddisk,ram .

########CPU Information########
Machine Uptime Information:
 uptime
 10:25:06 up 16:50,  1 user,  load average: 5.84, 5.65, 5.54
Command Description:
<current time> <Uptime since> <hour:min>
Number of users logged in to machine
load average <for last one minute>,<for last 5 minutes>,<for last 15 minutes>
________________________________________________________________
Terminal Information:
 w
 10:25:07 up 16:50,  1 user,  load average: 5.84, 5.65, 5.54
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/1    node1           10:18    5.00s  3.61s  0.64s -bash
________________________________________________________________
 cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
stepping        : 10
cpu MHz         : 2660.056
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts mmx fxsr sse sse2 ss syscall lm constant_tsc pni ds_cpl cx16 xtpr lahf_lm
bogomips        : 6660.87
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
stepping        : 10
cpu MHz         : 2660.056
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts mmx fxsr sse sse2 ss syscall lm constant_tsc pni ds_cpl cx16 xtpr lahf_lm
bogomips        : 6660.87
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

________________________________________________________________
Top 10 C.P.U Consumer Processes:
 ps -auxf | sort -nr -k 3 | head -10
root     13857 59.5  0.0  65532   972 pts/1    R+   10:25   0:01              \_ ps -auxf
root     13858 27.5  0.0 109112   664 pts/1    S+   10:25   0:00              \_ sort -nr -k 3
oracle   13842 21.7  0.1 1732792 13488 ?       Rs   10:25   0:02 oracleoracle (LOCAL=NO)
oracle   13860 20.0  0.1  74384 13508 ?        R    10:25   0:00      \_ /u01/app/oracle/111/bin/tnslsnr LISTENER -inherit
oracle   13845 18.4  0.1 1732280 10772 ?       Rs   10:25   0:01 oracleoracle (LOCAL=NO)
oracle   13855 14.6  0.0      0     0 ?        R    10:25   0:00  \_ [tnslsnr]
oracle   13840 13.4  0.1 1732280 12084 ?       Rs   10:25   0:01 oracleoracle (LOCAL=NO)
oracle   10080 10.9  3.4 1740824 279276 ?      Ss   09:29   6:10 oracleoracle (LOCAL=NO)
root     13859 10.5  0.0      0     0 pts/1    R+   10:25   0:00              \_ [bash]
root     13847  8.4  0.0   8680   944 ?        Ss   10:25   0:00      \_ /bin/sh -c /usr/sbin/ntpdate -b 10.180.23.51 >/dev/null 2>&1
________________________________________________________________
########Memory Information########
 cat /proc/meminfo
MemTotal:      8126860 kB
MemFree:       4766148 kB
Buffers:        375952 kB
Cached:        2203984 kB
SwapCached:          0 kB
Active:        1376116 kB
Inactive:      1319760 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8126860 kB
LowFree:       4766148 kB
SwapTotal:     2555888 kB
SwapFree:      2555888 kB
Dirty:             480 kB
Writeback:           0 kB
AnonPages:      116040 kB
Mapped:         453748 kB
Slab:           387024 kB
PageTables:      23384 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   6619316 kB
Committed_AS:  2439244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      1516 kB
VmallocChunk: 34359736843 kB
________________________________________________________________
 free -m
             total       used       free     shared    buffers     cached
Mem:          7936       3280       4656          0        367       2152
-/+ buffers/cache:        760       7175
Swap:         2495          0       2495
________________________________________________________________
Top 10 Memory Consumer Processes:
 ps -auxf | sort -nr -k 4 | head -10
oracle   10080 10.9  3.4 1740824 279304 ?      Ss   09:29   6:10 oracleoracle (LOCAL=NO)
oracle   10067  5.8  1.8 1742880 149684 ?      Ss   09:28   3:17 oracleoracle (LOCAL=NO)
oracle    9853  0.2  1.5 1735532 123392 ?      Ss   09:25   0:09 ora_smon_oracle
oracle    9857  0.5  0.9 1738836 79624 ?       Ss   09:26   0:18 ora_mmon_oracle
oracle    9994  4.8  0.8 1741860 66148 ?       Ss   09:27   2:46 oracleoracle (LOCAL=NO)
oracle    9847  0.5  0.5 1737024 42632 ?       Ss   09:25   0:19 ora_dbw0_oracle
oracle    9845  0.1  0.5 1732800 44440 ?       Ss   09:25   0:05 ora_mman_oracle
oracle   10214  0.7  0.5 1734396 41364 ?       Ss   09:31   0:25 ora_cjq0_oracle
oracle   10063  0.1  0.5 1740824 44272 ?       Ss   09:28   0:03 oracleoracle (LOCAL=NO)
oracle   10050  0.1  0.5 1740836 42716 ?       Ss   09:28   0:05 oracleoracle (LOCAL=NO)
________________________________________________________________
########Disk Information########
Disk Utilization
 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       13G  9.5G  2.4G  80% /
/dev/hda1              99M   20M   75M  21% /boot
tmpfs                 3.9G  1.6M  3.9G   1% /dev/shm
/dev/hdb1             5.0G  414M  4.3G   9% /mnt
/dev/hdd               30G  5.5G   23G  20% /data
10.180.23.194:/dump    29G  1.9G   26G   7% /emdump
________________________________________________________________
Disk Partition Information
 cat /proc/partitions
major minor  #blocks  name

   3     0   15728640 hda
   3     1     104391 hda1
   3     2   15623212 hda2
   3    64    5242880 hdb
   3    65    5237158 hdb1
  22    64   31457280 hdd
 253     0   13565952 dm-0
 253     1    2031616 dm-1
________________________________________________________________

zaxxon · August 6, 2010, 4:40am

Could you post the output of a

vmstat 1 10

when it is currently having this issue?

pinga123 · August 6, 2010, 5:16am

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 5831512  96952 1773232    0    0    35    27   84   43 31 47 21  1  0
 8  1      0 5830232  96956 1773228    0    0     0   200  252  125 35 49 14  1  0
 6  1      0 5829472  96996 1773236    0    0     0   212  366  182 30 68  1  0  0
 5  0      0 5828984  97056 1773176    0    0     0   168  282  164 39 61  0  0  0
 6  0      0 5827488  97056 1773260    0    0     0     0  264   90 35 65  0  0  0
 5  0      0 5826140  97056 1773260    0    0     0    48  263  108 35 65  0  0  0
 4  0      0 5820800  97064 1773252    0    0     0    12  353  139 41 57  2  0  0
 7  0      0 5818164  97064 1773252    0    0     0     0  376  178 41 57  2  0  0
 5  0      0 5820180  97068 1773260    0    0     0   180  325  147 35 63  1  0  0
 1  0      0 5827320  97068 1773260    0    0     0    12  322  146 28 33 39  0  0

---------- Post updated at 04:16 AM ---------- Previous update was at 04:15 AM ----------

I dont have a much information on when this problem occurred but whenever i try to execute any command or take a putty session of the machine it takes too much time to execute it.

zaxxon · August 6, 2010, 9:55am

This systems is CPU bound:

 6  1      0 5829472  96996 1773236    0    0     0   212  366  182 30 68  1  0  0
 5  0      0 5828984  97056 1773176    0    0     0   168  282  164 39 61  0  0  0
 6  0      0 5827488  97056 1773260    0    0     0     0  264   90 35 65  0  0  0
 5  0      0 5826140  97056 1773260    0    0     0    48  263  108 35 65  0  0  0
 4  0      0 5820800  97064 1773252    0    0     0    12  353  139 41 57  2  0  0
 7  0      0 5818164  97064 1773252    0    0     0     0  376  178 41 57  2  0  0
 5  0      0 5820180  97068 1773260    0    0     0   180  325  147 35 63  1  0  0

I made the run-queue for kernel threads bold. According to that you can see that there is all CPU-time used up for user and system while at same time idle is zero.
The top ten of your CPU-using processes might not help since there could be loads of small processes, that use only a bit of CPU-time but all together a big amount.
You might want to check top for a longer listing when this ps aux is not running and also you might check with a ps piped into awk for example to count how many processes are using how much CPU, as a summary.

You can try something like

ps -eo pcpu,comm| awk '{_[$2]+=$1} END{for(a in _)print _[a],a| "sort"}'

It is not very exact but it adds the CPU-time in % of the same commands up.

It could be that the box has so much trouble because there is something uneffectively written/configured in the software or if everything is allright, it might need more CPUs. Just a guess so far - you have to monitor and research more.

thegeek · August 6, 2010, 11:13am

What is your uptime shows ?

pinga123 · August 17, 2010, 12:50am

uptime
 10:25:06 up 16:50,  1 user,  load average: 5.84, 5.65, 5.54

thegeek · August 17, 2010, 4:25am

How many CPUs you have ?

If it is only 1, then you have to do something for process performance, like increasing the processor, or restricting unwanted process in that machine and make it exclusive for your purposes.

pinga123 · August 17, 2010, 5:52am

I Have 2 cpus.

 #cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
stepping        : 10
cpu MHz         : 2660.056
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts mmx fxsr sse sse2 ss syscall lm constant_tsc pni ds_cpl cx16 xtpr lahf_lm
bogomips        : 6660.87
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
stepping        : 10
cpu MHz         : 2660.056
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts mmx fxsr sse sse2 ss syscall lm constant_tsc pni ds_cpl cx16 xtpr lahf_lm
bogomips        : 6660.87
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

zaxxon · August 17, 2010, 9:31am

Did you check if there is maybe a big amount small processes (CPU usage time wise), that sum up to a big CPU usage time? Could you identify any process that eats up all the time?
Did you sum up the CPU time of all the oracle processes for example?

pinga123 · August 17, 2010, 11:45pm

Can u please guide me how would i do so?

zaxxon · August 18, 2010, 4:16am

I already wrote in a previous post a line of code how to sum up. Also you can grep -c for some pattern like "oracle" in the output of ps aux to see how many processes are there. I mostly wrote everything I would try. Check my previous posts.