I've been working with an HP-UX system (RP5400 Series PA-RISC server) for about a year that hosts some middleware. The middleware sits between an Oracle DB (on another box) and the client applications running on about 800 PCs. From the beginning, I've noticed that 'top' reports between 0.0% and 10% Idle during the day. Seeing that this is the first big production Unix server I've ever worked with, I believe that these Idle times are WAY too low. However, the application vendor told us that 0% idle should be OK as long as it doesn't stay there. Another support person at the application vendor, however, told me that we should be seeing between 30-60% Idle during the day! My gut feeling is that if it drops below 20% consistently, there is probably a resource issue. Here is a typical 'top' screen from our system in the middle of the day:
I've been monitoring with 'sar' as well and keeping a month's worth of 'sar' output in 15 minute intervals. A lot of what I see with 'sar' seems to reflect what I see with 'top', so I believe we are being pegged for CPU time by the user load. WIth all of this said, my base question is... what IS a reasonable Idle value/range on a Unix box? I've never actually seen this stated anywhere and I imagine it probably varies, but there should be some cushion, shouldn't there?
We have an oracle database server. It's an rp5400 with 4 cpu's. Idle time is in the 90's. These are powerful cpu's. Zero idle on that system would probably indicate run-away processes. Or incredible amounts of computation. Like finding the next mersenne prime or something.
I would say there doesn't have to be a cushion. We have boxes that run all day at 100% CPU and if the boxes can meet the performance requirements set forth by the buisness, then it just means we are not wasting any funds on hardware that is just sitting idle.
Here is a good senario, lets say I add one cpu to the machine that is running at 100% all day thus effectively doubling the cpu resources. Now I see 50% idle vs 100% idle CPU. This may be bad, because this may mean that I just wasted money on a CPU. Now other resources such as memory or I/O are now to slow for that much CPU.
It can also go the other way, you might still see 100% CPU, but that means that you never had enough CPU to utilize the memory and I/O anyhow. See there is a balance.
You do have alot of processes running and even sleeping processes use resources so I would not be too worried about this unless there are performance issues, than I would make sure that when adding more CPU I have suffecient memory and I/O to utilize the CPU.
Obviously there is alot more to it than this, such as what the processes running are doing etc. and might not apply to you, but it is something to think about.
[root@unix_srv1 root]# swapinfo -t
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 4149496 44808 99% 0 - 1 /dev/vg00/lvol2
dev 8384512 1000272 7384240 12% 0 - 2 /dev/vg03/lvol1
dev 8384512 1029924 7354588 12% 0 - 2 /dev/vg03/lvol2
reserve - 12118200 -12118200
memory 13059784 2880056 10179728 22%
total 34023112 21177948 12845164 62% - 0 -
[root@unix_srv1 root]# vmstat 1 6
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
7 0 0 2307190 249357 584 484 11 15 0 0 52 8106 55961 6824 52 16 31
7 0 0 2307190 247209 383 233 50 0 0 0 231 4782 53993 5288 80 19 1
7 0 0 2307190 249481 558 441 40 0 0 0 184 4855 58674 5312 71 29 0
7 0 0 2307190 246198 701 651 35 0 0 0 147 5197 63927 5689 76 24 0
7 0 0 2307190 246198 877 686 29 0 0 0 117 5172 63334 5480 80 20 0
7 0 0 2307190 247212 834 587 23 0 0 0 93 5453 63170 5747 66 26 8
Hmmm... the CPU idle is going all over the place right now. I ran 'vmstat' a few times and idle has ranged from 0-100. The swap usage is steadily increasing as system performance is dropping.
Your problem is physical memory, not swap space. Your scan rate (sr) is high (ideally should be 0), so the system is trying to find memory pages to free. But your reclaims (re) is also high. As fast as pages are placed on the free list, they are reclaimed.
That confirms my suspicions. Thanks! We are supposed to be getting the next model up from the RP5400 which allows for 24 Gigs of RAM. But, from the looks of it, that may not be enough. Scary. I think we'll have to look into getting our vendor to tune their code a little better. The multiple, per-process lookup tables are probably the worst of it.
How did you know about our work on the mersenne prime? ;p
Actually, the Oracle DB that our middleware utilizes is on an RX5670 (Itanium) box and that box is between 80-100% idle. So I think that Oracle is, expectedly, well behaved and the Itanium performs well. I've also been looking at the %CPU figure for processes and nothing seems to be grabbing a lot of CPU. What we do have is a lot of processes (over half [800+] are for the client PC connections). But they still only utilize a small portion of CPU (0.02% typically) so I'm not seeing anything that adds up to the high user CPU percentage figures I'm seeing. I've used 'ps -eo user,args,pcpu' there don't appear to be any major CPU hogs. The largest processes take up about 14-20% CPU but only briefly throughout the day.
Now that I look at things a little deeper though... Where I think the issue lies is in the architecture of the middleware and how it uses RAM. Looking at the RAM used by all process images for the middleware user account, it is constantly over the amount of physical RAM in the box. The vendor was particularly freaked out when they realized how large our client data set is which affects the size of lookup tables that they run for each process. It looks like we have (at this moment):
857 processes using 6M each ~10 Gigs
159 processes at 1.1M each ~180 Megs
43 processes at about 150M each ~6.5 Gigs
14 processes at about 111M each ~1.2 Gigs
Total = ~17 to 18 Gigs
Our box is maxed at 16 Gigs of physical RAM right now. So it would appear that we must be swapping a bit. When I tried a total of the vsz output of ps for the application user account, I saw 36 Gigs worth of process RAM usage. So... maybe the CPU user/sys/idle percentages are more a reflection of RAM usage? Does this sound like a possibility? I've already checked for i/o bottlenecks on our disk array and have cleared that as a possible problem point.
I believe we've populated the box's memory slots fully, so I don't think there is much we can do there until we get a bigger box.
We are supposed to be getting the next model up from the RP5400 which allows for 24 Gigs of RAM. But, from the looks of it, that may not be enough. Scary. I think we'll have to look into getting our vendor to tune their code a little better. The multiple, per-process lookup tables are probably the worst of it.