Hi
We have an AIX5.3 server with application which is written in C. We are facing server (lpar) hangs intermediately. If we open new telnet window prompts for user and takes hell of a time to authenticate, not only that if we run ps -aef then also it takes lot of time. surprisingly there is no cpu bottleneck as cpu is almost 80% idle, there is o memory or paging, there are no errors on network card. The only thing we observed is there were so many errors of core dump for the application.
Kbps tps Kb_read Kb_wrtn time
585.2 14.8 288 17264 09:57:51
Adapter: xfers time
-------------------- --------------------------- ---------
bps tps bread bwrtn
sissas0 598.9K 14.7 9.8K 589.1K 09:57:51
Disks: xfers read write queue time
-------------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------
%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv
act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull
hdisk0 3.1 302.1K 7.9 7.5K 294.6K 1.5 4.5 2.6 7.9 0 0 6.5 5.4 1.9 7.9 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:57:51
hdisk1 2.8 296.9K 6.8 2.3K 294.6K 0.4 6.8 3.4 11.8 0 0 6.5 5.5 2.5 11.3 0 0 0.0 0.0 0.0 0.0 0.0 0.0 09:57:51
Kbps tps Kb_read Kb_wrtn time
369.8 68.2 64 11030 09:58:21
Adapter: xfers time
-------------------- --------------------------- ---------
bps tps bread bwrtn
sissas0 378.7K 68.2 2.2K 376.5K 09:58:21
Disks: xfers read write queue time
-------------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------
%tm bps tps bread bwrtn rps avg min max time fail wps avg min max time fail avg min max avg avg serv
act serv serv serv outs serv serv serv outs time time time wqsz sqsz qfull
hdisk0 14.9 190.9K 34.4 2.0K 188.9K 0.4 4.2 0.0 0.0 0 0 34.0 19.2 1.5 8.5 0 0 22.8 0.0 0.0 0.5 0.3 10.6 09:58:21
hdisk1 14.5 187.8K 33.8 136.5 187.6K 0.0 17.1 0.0 0.0 0 0 33.8 18.4 1.3 4.6 0 0 20.8 0.0 0.0 0.5 0.3 10.2 09:58:21
"vmstat output"
System configuration: lcpu=2 mem=5376MB
kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
3 0 0 1213172 14247 30 0 0 0 0 0 3710 115624 11911 53 31 13 3 09:57:13
2 0 0 1213960 13119 65 256 0 0 0 0 1819 223286 7149 34 59 5 3 09:57:14
2 0 0 1224245 2788 46 704 0 0 723 1716 3080 938143 11432 52 43 4 0 09:57:15
4 0 0 1227928 1085 16 32 0 0 2081 4502 2285 96142 15999 28 66 5 0 09:57:16
2 1 0 1236307 1531 0 13 0 0 9000 24997 1031 14441 3713 9 37 53 1 09:57:17
0 0 0 1224268 14239 12 1 0 0 508 1319 3107 56542 12801 24 29 46 1 09:57:18
2 0 0 1236983 2272 0 0 0 0 923 3437 2748 54970 10001 24 22 54 0 09:57:19
1 0 0 1217684 21522 45 737 0 0 256 1375 3546 155013 13750 31 34 34 0 09:57:20
4 0 0 1214541 24604 8 64 0 0 0 0 4082 86059 17777 30 44 21 5 09:57:21
0 1 0 1211946 27162 0 36 0 0 0 0 4236 36690 14955 27 18 47 7 09:57:22
1 0 0 1211769 27338 0 0 0 0 0 0 3716 26524 14050 22 20 58 0 09:57:23
1 0 0 1211963 27143 0 0 0 0 0 0 3645 47245 14894 24 23 53 0 09:57:24
0 0 0 1212044 27029 17 0 0 0 0 0 5092 42021 18800 34 26 39 0 09:57:25
3 0 0 1212056 27026 1 0 0 0 0 0 7440 70306 29180 36 26 38 0 09:57:26
0 0 0 1212064 27018 0 0 0 0 0 0 4734 37139 18375 28 25 47 0 09:57:27
2 0 0 1212146 26936 0 15 0 0 0 0 4068 22927 14339 25 18 57 0 09:57:28
0 0 0 1212149 24933 0 1984 0 0 0 0 4928 34577 17412 36 25 39 0 09:57:29
0 0 0 1212147 24936 0 0 0 0 0 0 5531 32206 19008 40 18 42 0 09:57:30
0 0 0 1212149 24933 0 0 0 0 0 0 3785 26694 13758 23 22 55 0 09:57:31
0 0 0 1212141 24940 1 0 0 0 0 0 3908 21671 14132 23 15 62 0 09:57:32
kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 0 0 1212152 24933 0 0 0 0 0 0 3858 37512 14007 24 22 54 0 09:57:33
2 0 0 1212697 24379 10 1 0 0 0 0 3921 28358 13789 26 18 56 1 09:57:34
1 0 0 1212706 24370 0 0 0 0 0 0 4285 31408 15405 35 17 49 0 09:57:35
2 0 0 1212711 24364 1 0 0 0 0 0 3454 29837 12735 52 22 27 0 09:57:36
0 0 0 1212772 24302 0 0 0 0 0 0 4121 27853 15377 28 16 56 0 09:57:37
6 0 0 1212978 24095 0 1 0 0 0 0 3263 26979 14024 35 18 47 0 09:57:38
1 0 0 1213111 23960 2 0 0 0 0 0 3877 31711 13522 24 16 60 0 09:57:39
1 0 0 1213119 23952 0 0 0 0 0 0 3350 24652 13280 20 20 60 0 09:57:40
1 0 0 1213191 23880 0 0 0 0 0 0 3833 21134 13018 23 15 62 0 09:57:41
1 0 0 1213193 23874 0 0 0 0 0 0 4033 30411 14517 29 22 49 0 09:57:42
1 0 0 1213195 23868 1 0 0 0 0 0 4117 25976 14614 29 18 52 0 09:57:43
1 0 0 1213199 23828 55 0 0 0 0 0 3979 38583 14675 27 22 50 1 09:57:44
0 0 0 1213316 23711 0 0 0 0 0 0 3930 90174 13890 25 18 57 0 09:57:45
1 0 0 1213311 23713 0 0 0 0 0 0 3793 27332 13836 29 21 50 0 09:57:46
0 0 0 1213362 23654 5 0 0 0 0 0 3891 31158 13841 28 16 56 0 09:57:47
2 0 0 1213537 23480 0 1 0 0 0 0 3290 54229 11802 28 21 51 0 09:57:48
0 0 0 1213579 23437 0 0 0 0 0 0 3894 30331 14546 23 16 61 0 09:57:49
0 0 0 1213423 23589 2 1 0 0 0 0 4031 32409 15190 32 22 45 0 09:57:50
1 0 0 1211081 25922 8 0 0 0 0 0 4285 24381 15064 29 17 54 1 09:57:51
3 0 0 1212575 24427 0 0 0 0 0 0 4033 27862 14775 28 19 53 0 09:57:52
kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
0 0 0 1211471 25528 24 0 0 0 0 0 4032 26727 15380 25 18 57 0 09:57:53
0 0 0 1211471 25528 0 0 0 0 0 0 3678 20203 12995 23 15 63 0 09:57:54
0 0 0 1211483 25514 0 0 0 0 0 0 3284 32100 14016 21 21 58 0 09:57:55
2 0 0 1212630 24364 0 0 0 0 0 0 2705 23693 11065 18 13 69 0 09:57:56
1 0 0 1212788 24210 0 0 0 0 0 0 4169 32903 15619 28 24 48 0 09:57:57
0 0 0 1213009 23986 14 1 0 0 0 0 1806 12979 7245 11 8 80 0 09:57:58
2 0 0 1213014 23981 0 0 0 0 0 0 949 12599 4039 10 13 78 0 09:57:59
0 0 0 1213116 23878 0 0 0 0 0 0 2181 13214 7538 19 9 72 0 09:58:00
0 0 0 1213232 23761 0 0 0 0 0 0 2380 21901 9532 19 17 63 0 09:58:01
0 0 0 1213277 23716 0 0 0 0 0 0 1350 9198 4559 15 6 78 0 09:58:02
0 0 0 1213284 23708 0 23 0 0 0 0 1517 14331 5235 14 15 61 10 09:58:03
0 0 0 1213410 23577 0 60 0 0 0 0 4688 29454 16426 38 22 26 14 09:58:04
2 1 0 1213430 23555 0 222 0 0 0 0 3169 28903 11793 24 21 39 16 09:58:05
1 0 0 1213428 23557 0 176 0 0 0 0 3979 21043 13045 26 18 36 20 09:58:06
1 0 0 1213429 23555 0 10 0 0 0 0 4348 29102 14229 24 25 49 3 09:58:07
0 1 0 1213437 23547 0 1 0 0 0 0 3016 45648 9699 23 14 58 5 09:58:08
0 0 0 1212750 24232 0 0 0 0 0 0 1601 23978 8712 20 16 64 0 09:58:09
0 0 0 1212478 24503 0 0 0 0 0 0 628 43858 4317 15 19 67 0 09:58:10
1 0 0 1213179 23797 1 0 0 0 0 0 2568 50097 9130 28 20 51 0 09:58:11
0 0 0 1214633 22342 0 0 0 0 0 0 4052 23891 13778 31 17 52 0 09:58:12
kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
0 0 0 1217873 19102 0 0 0 0 0 0 3865 31917 13392 31 18 51 0 09:58:13
0 0 0 1217676 19289 10 0 0 0 0 0 3464 24873 12013 41 17 42 0 09:58:14
2 0 0 1218242 18683 0 32 0 0 0 0 3843 35720 14061 29 18 53 0 09:58:15
0 0 0 1218247 18678 0 0 0 0 0 0 3400 27049 13059 21 19 60 0 09:58:16
1 0 0 1218229 18696 0 0 0 0 0 0 4125 26827 16139 24 16 60 0 09:58:17
1 0 0 1218466 18458 0 1 0 0 0 0 5322 76074 20456 35 26 38 0 09:58:18
0 0 0 1218462 18462 0 0 0 0 0 0 3161 31550 12481 21 15 65 0 09:58:19
3 0 0 1218486 18435 0 0 0 0 0 0 5019 48378 19111 31 24 45 0 09:58:20
2 0 0 1218278 18644 0 0 0 0 0 0 5803 68820 21453 38 22 40 0 09:58:21
2 0 0 1218271 18650 0 0 0 0 0 0 4946 45197 18728 31 23 46 0 09:58:22
3 0 0 1218326 18596 0 0 0 0 0 0 3750 26586 15387 24 15 61 0 09:58:23
2 0 0 1218544 18378 0 0 0 0 0 0 2809 29751 12687 26 18 56 0 09:58:24
0 0 0 1219093 17826 0 0 0 0 0 0 2700 25927 11668 26 14 60 0 09:58:25
1 0 0 1219730 17185 0 0 0 0 0 0 2939 32015 13081 29 21 50 0 09:58:26
1 0 0 1219738 17177 0 0 0 0 0 0 2765 21737 11996 23 13 64 0 09:58:27
0 0 0 1219947 16967 0 5 0 0 0 0 2824 27244 11965 30 20 50 0 09:58:28
1 0 0 1219943 16967 5 2 0 0 0 0 4182 37190 16750 39 18 43 0 09:58:29
3 0 0 1220099 16811 0 0 0 0 0 0 5989 44857 22535 38 25 37 0 09:58:30
2 0 0 1220931 15978 0 0 0 0 0 0 5692 40823 21709 44 21 35 0 09:58:31
0 0 0 1221438 15470 8 0 0 0 0 0 5537 56461 21453 36 27 37 0 09:58:32
kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 0 0 1220941 15966 1 0 0 0 0 0 5946 49800 21525 43 23 34 0 09:58:33
2 0 0 1221772 15135 0 0 0 0 0 0 6246 58446 23431 37 23 40 0 09:58:34
6 0 0 1221608 15304 0 0 0 0 0 0 4981 77505 19651 38 28 35 0 09:58:35
3 0 0 1221741 15158 18 1 0 0 0 0 4802 229972 18775 36 31 33 0 09:58:36
3 0 0 1222282 14616 0 2 0 0 0 0 4094 173709 16648 35 30 34 0 09:58:37
2 0 0 1222632 14249 13 4 0 0 0 0 3331 140823 13253 31 26 40 2 09:58:38
1 0 0 1223227 13652 0 0 0 0 0 0 3093 55930 12104 30 20 50 0 09:58:39
0 0 0 1223519 13357 0 6 0 0 0 0 5113 39177 18264 35 19 45 1 09:58:40
1 0 0 1223559 13317 0 6 0 0 0 0 6152 58235 23403 36 27 37 0 09:58:41
0 0 0 1223652 13222 9 0 0 0 0 0 2011 19324 7381 21 10 69 0 09:58:42
0 0 0 1223876 12999 0 0 0 0 0 0 2494 40136 10229 23 19 58 0 09:58:43
1 0 0 1224076 12799 0 0 0 0 0 0 5877 46555 21618 35 20 45 0 09:58:44
0 0 0 1222992 13879 0 2 0 0 0 0 5951 49562 23099 34 28 38 0 09:58:45
0 0 0 1223136 13663 74 10 0 0 0 0 5127 32905 18769 30 18 51 0 09:58:46
1 0 0 1222404 14397 0 0 0 0 0 0 5520 84529 21312 33 27 39 0 09:58:47
0 0 0 1222452 14347 0 1 0 0 0 0 5349 89317 20386 29 22 48 0 09:58:48
0 0 0 1222120 14652 26 0 0 0 0 0 3896 32365 16100 24 21 54 1 09:58:49
1 0 0 1222142 14618 12 0 0 0 0 0 6049 48556 22919 34 21 45 0 09:58:50
1 0 0 1222304 14454 0 0 0 0 0 0 6668 73227 26926 36 30 34 0 09:58:51
1 0 0 1221187 15567 8 2 0 0 0 0 6228 94519 24593 36 29 34 0 09:58:52
kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa hr mi se
1 0 0 1222168 14582 0 0 0 0 0 0 5800 165168 23250 36 30 34 0 09:58:53
2 0 0 1220658 16094 0 0 0 0 0 0 4652 214673 18782 34 35 30 0 09:58:54
0 0 0 1220683 16067 0 0 0 0 0 0 4089 49429 16681 25 18 57 0 09:58:55
0 0 0 1220683 16067 0 0 0 0 0 0 5573 43567 22323 29 25 46 0 09:58:56
1 0 0 1220729 16008 22 0 0 0 0 0 3380 25609 12853 24 13 62 1 09:58:57
3 0 0 1220932 15803 0 1 0 0 0 0 3531 41507 13943 24 21 54 0 09:58:58
4 0 0 1218686 18047 0 0 0 0 0 0 3117 22044 11402 26 14 60 0 09:58:59
2 0 0 1215429 21302 0 3 0 0 0 0 3148 27815 12031 25 21 54 0 09:59:00
3 0 0 1215424 21307 0 0 0 0 0 0 4444 27695 16145 30 16 54 0 09:59:01
1 0 0 1216411 20314 0 0 0 0 0 0 3855 33125 14212 34 21 44 0 09:59:02
1 0 0 1216850 19875 0 0 0 0 0 0 4572 51358 17211 34 18 48 0 09:59:03
2 0 0 1217038 19684 16 0 0 0 0 0 3853 31791 14190 36 21 43 0 09:59:04
2 0 0 1216973 19656 3 32 0 0 0 0 5064 40493 19548 42 19 39 0 09:59:05
0 0 0 1217087 19374 0 192 0 0 0 0 5531 45886 22204 42 25 33 0 09:59:06
1 1 0 1217172 19120 0 214 0 0 0 0 6359 47394 24709 42 24 25 10 09:59:07
2 1 0 1217169 18952 12 249 0 0 0 0 4963 39831 19332 39 26 21 13 09:59:08
2 1 0 1217172 18779 0 228 0 0 0 0 6391 44199 24828 41 25 24 10 09:59:09
2 0 0 1217357 18437 0 191 0 0 0 0 6586 62367 26637 46 28 26 0 09:59:10
1 0 0 1217362 18257 8 128 0 0 0 0 5587 59916 24099 39 27 34 0 09:59:11
0 0 0 1217441 17991 0 224 0 0 0 0 3815 26898 16012 34 17 49 0 09:59:12
please suggest.