Hi,
I am getting a high load average, around 7, once an hour. It last for about 4 minutes and makes things fairly unusable for this time.
How do I find out what is using this. Looking at top the only thing running at the time is md5sum.
I have looked at the crontab and there is nothing running at this time, so is there a number of hidden crontabs that I don't know about?
Any help would be greatly appreciated.
Running WHM/Cpanel on CentOS
Thanks
Matt
There's all the user crontabs of course, also check for filenames under /etc/cron.hourly, these will as the name implies get run hourly. This doesn't rule out some script somewhere that does this of its own volition either, just a "while true ; do sleep 3600 ; hog_system ; done"
Depending on your system your version of ps may be able to print a tree of what created what process (or at least what process belongs to what process), potentially helpful for tracking down what's spawning the monster. Try ps -ejH
I wouldn't expect a high load average to lag the system, Linux is pretty good at keeping responsive despite high CPU load, but if something's hogging disk access -- like many instances of md5sum could -- that'd definitely slow things down by turfing useful things out of cache.
excellent, found the problem using the ps command.
I recently signed up for a VPS with CentOS and CPanel/WHM. The exact same thing is happening on mine. Spikes every hour, on the hour. They last about 4 minutes and during that time there is piss poor system performance.
High IO Wait time. Memory levels are fine. No RAM swapping.
I monitored top during these spikes and the process that kept floating to the top was: md5sum
---------- Post updated at 03:42 AM ---------- Previous update was at 03:37 AM ----------
By the way, how do we fix what is going on? I am running "ps -ejH" but not sure what to look for. It is neat looking at the process tree! Sweet command.
You may find this useful.
"Top 10 CPU Utilized Processes"
ssh -Tqn $1 ps auxf | sort -nr -k 3 | head -10
"Top 10 Memory Utilized Processes"
ssh -Tqn $1 ps auxf | sort -nr -k 4 | head -10
ps -ejH
22019 22019 22019 ? 00:00:03 lfd
26292 22019 22019 ? 00:00:00 lfd
26293 22019 22019 ? 00:00:00 md5sum
ps auxf | sort -nr -k 3 | head -10
root 26293 0.5 0.0 2928 620 ? D 05:00 0:00 | \_ /usr/bin/md5sum --check /etc/csf/cs
f.tempint
root 28483 0.2 1.2 16816 12908 ? D 05:00 0:00 \_ lfd - checking directories
What is lfd and any ideas how to bring this under control?
You may find this helpful.
LFD Process Email Alerts - Web Hosting Talk
---------- Post updated at 04:06 AM ---------- Previous update was at 04:04 AM ----------
I can see its not taking your cpu as much as you are worried it is.
It is just 0.2% of your cpu and you shouldn't be worrying about it.
There is a bit more going on that what these snapshots are showing you. If you saw my top during that time, I had a constant iowait around 50-80% despite lfd/md5sum showing significantly less CPU utilization.
I think the root issue is the way the CSF firewall was configured out of the box. Every 3600 seconds (60 minutes), it's doing a "System Integrity Check". Here is what they have to say about it:
I am going to set it to 0 to see if this solves the immediate problem.
---------- Post updated at 06:04 AM ---------- Previous update was at 05:28 AM ----------
bingo! the iowait for this hour is back to normal levels! I'll keep an eye on it today.
Low CPU doesn't mean it's not thrashing the disk. In fact low CPU on something with no built-in delays like md5sum probably means it IS thrashing the disk.
Does LOW CPU necessarily has to translate to 'thrashing the disk' I don't that is right? It could be simple IO wait on a NFS!!!
For something that's not interactive and never sleeps, like md5sum, if it's not pinning the CPU, it's waiting for I/O.
Thrashing the network is better how? And that'd still be thrashing someone's disk, if not yours.