Dear Friends,
we are using HP-UX B.11.31 U ia64 HP-UX server. Can you check bellow the top command output whether can point out any abnormality. Becoz i suspect something wrong there,
Load averages: 2.40, 2.73, 2.99
711 processes: 287 sleeping, 424 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 2.74 12.1% 23.0% 62.8% 2.1% 0.0% 0.0% 0.0% 0.0%
12 2.26 25.7% 40.2% 30.0% 4.2% 0.0% 0.0% 0.0% 0.0%
13 2.09 19.4% 29.6% 45.3% 5.7% 0.0% 0.0% 0.0% 0.0%
14 2.20 13.4% 41.3% 39.1% 6.2% 0.0% 0.0% 0.0% 0.0%
15 2.30 23.4% 17.7% 52.8% 6.0% 0.0% 0.0% 0.0% 0.0%
16 2.65 16.8% 22.1% 59.4% 1.7% 0.0% 0.0% 0.0% 0.0%
24 2.53 23.4% 27.0% 47.7% 1.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 2.40 19.2% 28.6% 48.2% 4.0% 0.0% 0.0% 0.0% 0.0%
System Page Size: 4Kbytes
Memory: 19777212K (15964960K) real, 29382676K (21944844K) virtual, 6644196K free Page# 1/28
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
16 ? 2607 root 152 20 62400K 15372K run 4030:43 78.14 78.01 nfsd
13 ? 20489 cgi 152 20 3476M 3449M run 535:59 72.39 72.26 uu_ccc_sss
14 ? 22075 cgi 152 30 52400K 24696K run 0:13 48.37 36.44 xx_yyy_mmm
thanks !
---------- Post updated 05-12-10 at 08:54 AM ---------- Previous update was 05-11-10 at 03:47 PM ----------
sorry guys if above content is not clear.
Actually what I want to know is what is the nfsd process running under root. I guess it is something related to OS nfs processors.
But should it take so much of the processor resource , and why the nice is getting 28% of CPU.
Is these normal .... ?
Consider getting a copy of the HP Tuning and Performance book.
The snapshot from "top" taken out of context doesn't mean a lot.
The percentage shown in "top" for a single process such as "nfsd" (see "man nfsd" to read about the process) is the percentage of one CPU. You have many CPUs.
If you have HP "glance" this is a much better tool for snapshots.
There are many packages for recording historical performance information including unix "sar" and commercial packages from HP. An hour-by-hour view of server performance is more valuable than a snapshot.
The "top" output posted implies a severe shortage of memory and CPU. This is only an implication and we would need better information before making a recommendation.
Whether an apparent shortage actually matters would need detailed analysis of swap statistics and CPU waits. You will need something better than "top".
One of the many advantages of unix over rival Operating Systems is that a correctly tuned server can perform well when theoretically overloaded.
1 Like
thank you methyl,
yah I think there analyzing historic data is impotent , but I'm not in a position to do that now.
Can I know if system processors uses more that 50% of the avg CPU usage is it normal ?
I used vmstat few times .. I saw there are lot of page ins some time
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
7 1 0 5107229 908037 0 0 3 11 3 0 3 10297 726646 17695 13 20 67
7 1 0 5107229 903699 0 0 0 0 0 0 0 21909 485854 26542 22 25 53
8 1 0 4946624 911139 0 0 264 0 0 0 0 17315 509344 24555 22 25 53
8 1 0 4946624 913250 0 0 85 0 0 0 0 11699 414913 20652 20 10 70
5 1 0 4827443 912616 0 0 27 0 0 0 0 12459 471801 21813 20 21 59
5 1 0 4827443 911064 0 0 7 0 0 0 0 9586 438479 22304 20 9 71
5 1 0 4314344 909637 0 0 0 0 0 0 0 7860 457082 21690 20 21 59
2 0 0 4827662 1036017 0 0 0 0 0 0 0 3160 92948 1934 15 8 77
3 1 0 4936522 1036737 0 0 328 0 0 0 0 5155 112688 2187 17 12 71
3 1 0 4936522 1039376 0 0 107 0 0 0 0 3424 75236 1196 15 7 78
3 0 0 4445451 1039428 0 0 34 0 0 0 0 2759 114349 1359 16 9 75
3 0 0 4445451 1041994 0 0 9 0 0 0 0 2359 79628 885 15 9 76
And this is a one moment of glance output
Glance C.04.70.001 17:45:41 S02CJ091 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SARU U | 42% 43% 100%
Disk Util F F | 59% 26% 100%
Mem Util S SU U | 91% 91% 92%
Networkil U UR R | 85% 84% 87%
------------------------------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 8
User CPU % Thrd Disk Memory Block
Process Name PID Name ( 700% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
eps_ui_soap_ 23731 cgi 96.2 2 0.0 538.2mb 557.2mb PRI
glance 4191 cgi 50.7 1 0.0 17.3mb 26.2mb STRMS
uio_cgisam_s 20489 cgi 38.4 5 221.0 3.47gb 3.49gb PRI
nfsd 2607 root 36.2 67 229.0 15.2mb 55.2mb OTHER
dcs_audit_re 9082 cgi 19.1 1 0.0 12.2mb 30.9mb SOCKT
midaemon 4215 root 7.6 7 0.0 136.2mb 140.8mb SLEEP
vxfsd 96 root 6.0 57 69.8 33.6mb 37.8mb SLEEP
bptm 10956 root 4.7 1 71.0 9.7mb 9.7mb died
vhand 2 root 3.9 1 0.2 64kb 72kb SLEEP
uio_cgisam_s 20459 cgi 2.4 81 20.5 3.73gb 3.90gb SOCKT
even though I dont believe above data would not help much to analyze .. in case you notice some thing pls let me know.
thanks
The "glance" program has more options which help find bottlenecks.
Press ? to get the menu or start glance with the option "e.g. glance -t".
In your case these should help:
w (swap activity)
t (system tables) Need to see all pages of this.
m (memory)
B (Global waits)
N (NFS)
First impressions are that you are short of memory but the above glance options should show an overview of what is happening. If you have pseudo-swap configured this can skew the memory figures.
BTW. The vmstat would mean more if we knew the sampling interval (i.e. the command line you typed).
Thanks Methyl,
vmstat sampling rate is vmstat 5.
I have added galnce outputs as you mentioned , if you happen to notice anything kindly let me know.
glance -w
Glance C.04.70.001 10:57:14 S02CJ091 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SNNARU U | 61% 64% 100%
Disk Util F F | 23% 31% 97%
Mem Util S SU U | 92% 92% 93%
Swap Util U UR R | 83% 83% 87%
------------------------------------------------------------------------------------------------------------------------------------------------------
SWAP SPACE Users= 9
Swap Device Type Avail Used Priority
--------------------------------------------------------------------------------
/dev/vg00/lvol2 device 8.0gb 0mb 1
/dev/vg00/lvol13 device 3.9gb 0mb 2
pseudo-swap memory 38.0gb 29.5gb -1
Swap Available: 51104m Swap Used: 30258m Swap Util (%): 83 Reserved: 42450m
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
glance -t
Glance C.04.70.001 10:59:33 S02CJ091 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SNNARU U | 80% 80% 80%
Disk Util F F | 31% 31% 31%
Mem Util S SU U | 92% 92% 92%
Networkil U UR R | 84% 84% 84%
------------------------------------------------------------------------------------------------------------------------------------------------------
SYSTEM TABLES REPORT Users= 9
System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 8212 685 8 8
File Table (nfile) 2147483647 9349 0 0
Shared Mem Table (shmmni) 1024 31 3 3
Message Table (msgmni) 2014 3 0 0
Semaphore Table (semmni) 10070 52 1 1
File Locks (nflocks) 10000 213 2 2
Pseudo Terminals (npty) 512 0 0 0
Buffer Headers (nbuf) na 9961 na na
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
glance -m
Glance C.04.70.001 11:01:33 S02CJ091 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
Cpu Util S SN NRU U | 94% 97% 100%
Disk Util F FV | 37% 35% 37%
Mem Util S SU U | 93% 93% 93%
Networkil U UR R | 87% 87% 87%
------------------------------------------------------------------------------------------------------------------------------------------------------
MEMORY REPORT Users= 9
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 140583 548128 8841.6 9929.8 10982.2
Page In 135 254 8.4 4.6 2790.3
Page Out 0 4 0.0 0.0 0.0
KB Paged In 540kb 1016kb 33.9 18.4 135.6
KB Paged Out 0kb 16kb 0.0 0.2 0.7
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 140 273 8.8 4.9 8.8
VM Writes 0 0 0.0 0.0 0.0
Total VM : 31.7gb Sys Mem : 12.2gb User Mem: 19.4gb Phys Mem : 40.0gb
Active VM: 23.4gb Buf Cache: 1mb Free Mem: 2.7gb FileCache: 5.7gb
MemFS Blk Cnt: 0 MemFS Swp Cnt: 0
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
glance -B
Glance C.04.70.001 11:05:26 S02CJ091 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
Cpu Util S SN NARU U | 90% 79% 91%
Disk Util F FV | 23% 24% 30%
Mem Util S SU U | 94% 93% 94%
Networkil U UR R | 88% 88% 88%
------------------------------------------------------------------------------------------------------------------------------------------------------
GLOBAL WAIT STATES Users= 9
Procs/ Procs/
Event % Time Threads Blocked On % Time Threads
--------------------------------------------------------------------------------
IPC 0.3 128.41 11.0 Cache 0.0 7.45 0.6
Job Control 0.0 0.37 0.0 CDROM IO 0.0 0.00 0.0
Message 0.0 0.00 0.0 Disk IO 0.0 0.00 0.0
Pipe 0.2 93.95 8.1 Graphics 0.0 0.00 0.0
RPC 0.0 0.00 0.0 Inode 0.0 0.98 0.1
Semaphore 0.0 0.00 0.0 IO 0.4 158.23 13.6
Sleep 27.7 12297.15 1054.6 LAN 0.0 0.00 0.0
Socket 6.4 2850.47 244.5 NFS 0.0 0.00 0.0
Stream 0.0 16.68 1.4 Priority 0.2 70.05 6.0
Terminal 0.0 0.00 0.0 System 1.2 527.80 45.3
Other 63.5 28217.66 2420.0 Virtual Mem 0.0 2.21 0.2
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
glance -N
Glance C.04.70.001 11:07:23 S02CJ091 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SN NARU U | 84% 84% 87%
Disk Util F F | 39% 42% 76%
Mem Util S SU U | 93% 93% 93%
Networkil U UR R | 86% 86% 87%
------------------------------------------------------------------------------------------------------------------------------------------------------
NFS GLOBAL ACTIVITY Users= 9
Server (inbound) Client (outbound)
Current Cum Current Cum
--------------------------------------------------------------------------------
Read Rate 635.2 684.4 0.0 0.0
Write Rate 110.2 126.1 0.0 0.0
Read Byte Rate 1244.1 1198.9 0.0 0.0
Write Byte Rate 650.7 830.8 0.0 0.0
NFS Call Count 9898 60879 0 0
Bad Call Count 0 0 0 0
Service Time 4.87 21.67 0.00 0.00
Network Time na na 0.00 0.00
Read/Write Qlen na na 0 0
Idle biods na na na na
thanks !!!