Hi all,
I have application running on sun server T5440 4x8x1.4 GHz, 64 GB RAM, application running very slow though load average too low. when I install my application on another server SUN M3000 (One CPU 1x8x2.5GHz, 8GB RAM), application run smoothly.
Here is my server T5440 info:
bash-3.00# uname -a
SunOS 5.10 Generic_142909-17 sun4v sparc SUNW,T5440
Psrinfo -pv
bash-3.00# psrinfo -pv
The physical processor has 64 virtual processors (0-63)
UltraSPARC-T2+ (chipid 0, clock 1414 MHz)
The physical processor has 64 virtual processors (64-127)
UltraSPARC-T2+ (chipid 1, clock 1414 MHz)
The physical processor has 64 virtual processors (128-191)
UltraSPARC-T2+ (chipid 2, clock 1414 MHz)
The physical processor has 64 virtual processors (192-255)
UltraSPARC-T2+ (chipid 3, clock 1414 MHz)
/etc/system
set semsys:seminfo_semopm=10000
set semsys:seminfo_semmsl=10000
set semsys:seminfo_semmni=1024
set max_nprocs=50000
set msgsys:msginfo_msgmnb=2097152
set msgsys:msginfo_msgmni=2560
set msgsys:msginfo_msgtql=2560
set shmsys:shminfo_shmmax=1073741824
set shmsys:shminfo_shmmni=1000
set rlim_fd_cur=10000
Ipcs
bash-3.00# ipcs -a
IPC status from as of Wed Jun 13 12:31:01 ICT 2012
T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME
Message Queues:
q 62 0x1388 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:22
q 61 0x1387 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:22
q 60 0x1318 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:22
q 59 0x12a9 --rw-rw-rw- topx topx topx topx 0 0 2097152 9069 9071 10:25:38 10:25:38 8:36:22
q 58 0x100f --rw-rw-rw- topx topx topx topx 0 0 2097152 9069 9070 10:06:16 10:06:16 8:36:22
q 57 0xfa0 -Rrw-rw-rw- topx topx topx topx 0 0 2097152 9071 9069 10:25:38 10:25:38 8:36:22
q 56 0xf30 --rw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:36:20
q 55 0xc27 -Rrw-rw-rw- topx topx topx topx 0 0 2097152 0 0 no-entry no-entry 8:28:14
q 54 0xbb8 -Rrw-rw-rw- topx topx topx topx 0 0 2097152 172 9066 10:47:08 10:47:08 8:28:14
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
Semaphores:
s 104 0x9c4 --ra-ra-ra- topx topx topx topx 3 12:31:00 8:36:20
s 103 0xa8c --ra-ra-ra- topx topx topx topx 1 8:36:20 8:36:20
s 102 0xd48 --ra-ra-ra- topx topx topx topx 1 8:36:20 8:36:20
s 101 0xce4 --ra-ra-ra- topx topx topx topx 1 8:36:20 8:36:20
s 100 0x26de --ra-ra-ra- topx topx topx topx 8 10:19:34 8:28:14
s 99 0xc1c --ra-ra-ra- topx topx topx topx 1 8:36:20 8:28:14
s 98 0xbb8 --ra-ra-ra- topx topx topx topx 1 8:43:50 8:28:14
s 97 0xaf0 --ra-ra-ra- topx topx topx topx 500 8:36:20 8:28:14
s 96 0xa28 --ra-ra-ra- topx topx topx topx 500 8:36:20 8:28:14
s 95 0x7ee --ra-ra-ra- topx topx topx topx 1 8:36:20 8:28:14
s 94 0x7e4 --ra-ra-ra- topx topx topx topx 1001 8:36:20 8:28:14
s 93 0x7da --ra-ra-ra- topx topx topx topx 220 8:36:20 8:28:14
s 92 0x7d5 --ra-ra-ra- topx topx topx topx 200 12:05:15 8:28:14
s 91 0x7d0 --ra-ra-ra- topx topx topx topx 200 8:56:42 8:28:14
s 90 0xc80 --ra-ra-ra- topx topx topx topx 2 12:31:01 8:27:52 -a
Prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
9202 topx 1460M 893M sleep 59 0 0:16:37 0.2% integrity_check/1
12455 root 3880K 3600K cpu65 59 0 0:00:00 0.0% prstat/1
9080 topx 142M 130M sleep 59 0 0:00:52 0.0% intf_resp_low/1
9125 topx 102M 78M sleep 59 0 0:00:36 0.0% interface_vsc/1
9121 topx 88M 80M sleep 59 0 0:00:32 0.0% interface_tcu/1
9120 topx 89M 80M sleep 59 0 0:00:29 0.0% interface_rcm/1
9124 topx 87M 78M sleep 59 0 0:00:29 0.0% baplie_merge_v2/1
9122 topx 86M 78M sleep 59 0 0:00:28 0.0% interface_eih/1
9115 topx 88M 77M sleep 59 0 0:00:33 0.0% intf_batch_even/1
9114 topx 88M 77M sleep 59 0 0:00:39 0.0% intf_event/1
9117 topx 87M 78M sleep 59 0 0:00:32 0.0% interface_bxy/1
9118 topx 86M 78M sleep 59 0 0:00:29 0.0% interface_ppm/1
NPROC USERNAME SWAP RSS MEMORY TIME CPU
41 topx 334M 2237M 3.4% 0:30:17 0.3%
43 root 186M 195M 0.3% 0:14:45 0.0%
1 noaccess 152M 136M 0.2% 0:04:00 0.0%
1 smmsp 3688K 11M 0.0% 0:00:04 0.0%
1 lp 1208K 3880K 0.0% 0:00:00 0.0%
6 daemon 7264K 9944K 0.0% 0:00:26 0.0%
Total: 93 processes, 259 lwps, load averages: 0.40, 0.39, 0.39
Prstat -mL
bash-3.00# prstat -mL
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
9202 topx 0.9 0.0 0.0 0.0 0.0 0.0 99 0.0 2 0 5 0 integrity_ch/1
9080 topx 0.4 0.0 0.0 0.0 0.0 0.0 100 0.0 5 0 20 0 intf_resp_lo/1
9114 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 13 0 66 0 intf_event/1
17282 root 0.1 0.2 0.0 0.0 0.0 0.0 100 0.0 20 1 324 0 prstat/1
9117 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 1 70 0 interface_bx/1
9118 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 1 70 0 interface_pp/1
9115 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 15 0 80 0 intf_batch_e/1
9121 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_tc/1
9125 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_vs/1
9120 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_rc/1
9123 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 0 70 0 interface_vs/1
9124 topx 0.2 0.0 0.0 0.0 0.0 0.0 100 0.0 10 1 70 0 baplie_merge/1
9122 topx 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 8 0 56 0 interface_ei/1
9119 topx 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 8 0 56 0 interface_cp/1
9204 topx 0.1 0.0 0.0 0.0 0.0 0.0 100 0.0 15 1 1K 0 sys_check/1
259 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 13 2 98 7 in.mpathd/1
1 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 99 3 init/1
1337 noaccess 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 100 0 100 0 java/14
140 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 19 0 114 0 nscd/851
Total: 90 processes, 256 lwps, load averages: 0.40, 0.40, 0.39
Vmstat
bash-3.00# vmstat 1 10
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m3 in sy cs us sy id
0 0 0 87720712 60562856 173 645 199 44 44 0 0 0 0 0 0 886 2819 525 0 1 99
1 0 0 87563976 59743040 73 327 0 15 15 0 0 0 0 0 0 747 2831 439 0 1 99
0 0 0 87563656 59742808 61 306 0 15 15 0 0 0 0 0 0 770 2684 427 0 1 99
1 0 0 87563656 59742824 0 0 0 0 0 0 0 0 0 0 0 886 1302 525 0 1 99
1 0 0 87563656 59742816 62 305 0 15 15 0 0 0 0 0 0 722 2652 420 3 1 96
1 0 0 87563656 59742840 554 2777 0 139 139 0 0 0 0 0 0 959 9736 749 1 1 97
0 0 0 87563456 59742672 1155 5735 0 292 292 0 0 0 0 0 0 1113 17489 1145 1 3 96
1 0 0 87563656 59742840 0 0 0 0 0 0 0 0 0 0 0 845 1613 654 0 1 99
0 0 0 87563656 59742856 62 305 0 15 15 0 0 0 0 0 0 900 2647 526 0 1 99
0 0 0 87563656 59742856 62 304 0 15 15 0 0 0 0 0 0 683 2661 396 0 1 99
Mpstat
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 130 261 1 121 0 13 1 0 0 0 1 0 99
1 0 0 35 133 2 143 0 9 6 0 64 0 0 0 100
2 0 0 5 20 2 17 0 3 0 0 29 1 0 0 99
3 0 0 0 3 0 2 0 0 0 0 0 0 0 0 100
4 0 0 3 14 0 15 0 2 2 0 13 0 0 0 100
73 0 0 474330 77 3 0 7 0 2 0 2596 67 33 0 0
73 0 0 487701 37 0 0 7 0 8 0 2670 66 34 0 0
73 0 0 476377 14 0 0 8 0 1 0 2610 66 34 0 0
78 0 0 473950 11 1 0 7 0 13 0 2593 67 33 0 0
78 0 0 480562 9 0 0 7 0 4 0 2630 66 34 0 0
79 0 0 320989 14 1 0 8 0 0 0 1766 77 23 0 0
79 0 0 479148 9 0 0 7 0 8 0 2622 66 34 0 0
79 0 0 479426 13 0 0 7 0 2 0 2622 67 33 0 0
79 0 0 488949 9 0 0 8 0 13 0 2678 66 34 0 0
64 0 0 482229 15 2 0 8 0 7 0 2638 65 35 0 0
77 0 0 475571 14 1 0 7 0 8 0 2602 67 33 0 0
69 0 0 467795 22 1 0 7 0 40 0 2563 66 34 0 0
69 0 0 318623 9 1 0 7 0 8 0 1751 77 23 0 0
66 0 0 465179 92 1 2 8 0 62 0 2545 66 34 0 0
When my application running, some of CPU have IDLE state = 0 and hold the CPU while another CPU have IDLE state 100. I think this is my bottleneck but I cannot find any solution for solve that.
Netstat
bash-3.00# netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 244022 0 244022 0 0 0
nxge0 1500 TOPX-01-DUMMY1 TOPX-01-DUMMY1 3594863 0 5801650 0 0 0
nxge1 1500 TOPX-01-DUMMY2 TOPX-01-DUMMY2 160954 0 151230 0 0 0
iostat
bash-3.00# iostat -xn
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 0.1 1.5 30.8 0.0 0.0 1.8 9.0 0 0 md/d0
0.0 0.1 0.8 30.8 0.0 0.0 0.0 9.9 0 0 md/d1
0.0 0.1 0.8 30.8 0.0 0.0 0.0 9.8 0 0 md/d2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 15.2 0 0 md/d3
0.0 0.0 0.0 0.0 0.0 0.0 0.0 10.9 0 0 md/d4
0.0 0.0 0.0 0.0 0.0 0.0 0.0 19.4 0 0 md/d5
0.0 0.0 2.2 1.5 0.0 0.0 0.8 7.4 0 0 md/d6
0.0 0.0 1.1 0.7 0.0 0.0 0.0 8.5 0 0 md/d7
0.0 0.0 1.1 0.8 0.0 0.0 0.0 8.2 0 0 md/d8
0.0 0.1 0.4 0.6 0.0 0.0 7.6 10.0 0 0 md/d9
0.0 0.1 0.2 0.6 0.0 0.0 0.0 9.1 0 0 md/d10
0.0 0.1 0.2 0.6 0.0 0.0 0.0 7.8 0 0 md/d11
0.0 0.0 0.0 0.0 0.0 0.0 0.1 10.4 0 0 md/d12
0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.5 0 0 md/d13
0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.4 0 0 md/d14
0.1 0.5 2.1 32.3 0.0 0.0 0.0 9.9 0 0 c0t0d0
0.1 0.5 2.1 32.3 0.0 0.0 0.0 9.8 0 0 c0t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.9 0.9 125.3 123.1 0.0 0.1 0.0 34.2 0 2 c4t600144F0CC00C29A00004D1E456F0003d0
0.3 0.7 20.7 53.9 0.0 0.0 0.2 8.2 0 0 c4t600144F0CC00C29A00004D1E43880001d0
0.1 0.2 50.7 111.5 0.0 0.0 0.0 59.1 0 1 c4t600144F0CC00C29A00004D1E453E0002d0
Please help me find the bottleneck of my T5440 server?Thanks so much for all your help.