Bad performance but Low CPU loading?

There might be some problem with my server,
because every morning at 7, it's performance become bad with no DB extra deadlock.
But I just couldn't figure it out.
Please give me some advise, thanks a lot...

According to the CPU performace chart, Daily CPU loading Maximum: 42 %, Average:36%.

But I have a serious Report Delay for almost half hour.

Here is the "TOP", at the same time but the next day I collect.

Load averages: 9.60, 9.80, 6.67
682 processes: 626 sleeping, 56 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 7.23 44.5% 24.0% 22.0% 9.4% 0.0% 0.0% 0.0% 0.0%
1 11.97 38.7% 29.6% 21.3% 10.5% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 9.60 41.6% 26.8% 21.7% 9.9% 0.0% 0.0% 0.0% 0.0%
Memory: 628612K (144072K) real, 338672K (92712K) virtual, 2251204K free Page# 1/22
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 29917 UserA 245 22 4868K 532K run 0:04 14.44 11.38 REPORT_A
0 ? 19416 UserB 241 20 5740K 732K run 0:20 11.11 11.09 REPORT_C
0 ? 29772 UserA 245 22 4804K 464K run 0:03 11.81 9.43 REPORT_B
0 ? 1411 root -16 20 28128K 5736K run 365:05 5.09 5.08 midaemon
1 ? 26114 UserB 203 20 5224K 604K run 0:06 2.46 2.45 REPORT_F
0 ? 29539 UserB 191 20 5136K 516K run 0:00 2.42 2.21 REPORT_G
0 ? 18222 UserA 217 22 5708K 728K run 0:15 1.98 1.98 REPORT_C
0 ? 23358 UserA 216 22 5708K 736K run 0:08 1.94 1.93 REPORT_C_b
0 ? 28030 UserB 197 20 5136K 516K run 0:04 1.92 1.92 REPORT_G
0 ? 28263 UserB 194 20 5136K 516K run 0:04 1.92 1.92 REPORT_G
1 ? 23417 UserA 154 22 5708K 728K sleep 0:08 1.85 1.85 REPORT_C_b
0 ? 33 root 152 20 1856K 0K run 54:44 1.79 1.79 vxfsd
0 ? 23381 UserA 216 22 5708K 728K run 0:08 1.77 1.77 REPORT_C_b
0 ? 23361 UserA 214 22 5708K 728K run 0:07 1.76 1.76 REPORT_C
0 ? 29696 UserB 192 20 5136K 516K run 0:00 2.11 1.72 REPORT_G
0 ? 23415 UserA 215 22 5708K 728K run 0:08 1.72 1.72 REPORT_C_b
0 ? 25207 UserB 189 20 5140K 536K run 0:07 1.72 1.72 REPORT_D
0 ? 27030 UserB 188 20 4808K 488K run 0:04 1.70 1.70 REPORT_E
0 ? 25729 UserB 189 20 5140K 536K run 0:06 1.70 1.70 REPORT_D
0 ? 29471 UserB 189 20 4808K 488K run 0:00 1.77 1.68 REPORT_E
0 ? 25571 UserB 191 20 5140K 536K run 0:06 1.66 1.66 REPORT_D
0 ? 24909 UserB 188 20 5140K 536K run 0:07 1.66 1.66 REPORT_D
0 ? 25127 UserB 190 20 5140K 536K run 0:06 1.65 1.65 REPORT_D
0 ? 26399 UserB 188 20 4864K 540K run 0:04 1.65 1.65 REPORT_Z
0 ? 26532 UserB 154 20 4808K 488K sleep 0:04 1.65 1.65 REPORT_E
0 ? 26177 UserB 189 20 4864K 540K run 0:05 1.65 1.64 REPORT_Z
0 ? 23478 UserA 217 22 4840K 540K run 0:07 1.64 1.64 REPORT_E
0 ? 23350 UserA 213 22 5708K 728K run 0:07 1.63 1.63 REPORT_C
0 ? 28311 UserB 190 20 4808K 488K run 0:03 1.63 1.62 REPORT_E
0 ? 25255 UserB 188 20 5140K 536K run 0:06 1.62 1.62 REPORT_D
0 ? 26355 UserB 189 20 4864K 540K run 0:04 1.62 1.62 REPORT_Z

Maybe your crontab file kicks off something like a daily backup script?

I checked some backup job, they're set at other time. Such as 1:00,23:00...

"top" is not much of a tool for diagnosing performance issues.

Do you have "glance".
Do you have "sar".
What Database Engine is this?

First let's find out how much memory you have. The figures from "top" are rarely correct.

/usr/sbin/swapinfo -mt

About 5G memory. Using Informix DB. Thanks..

I still can't tell what's wrong with the Server,
maybe midaemon process costs too much CPU?
Please help...thank you very much!

B3692A GlancePlus C.03.50.00    15:42:38  MyServer 9000/800                                                        Current  Avg  High
------------------------------------------------------------------------------------------------------------------------------------
CPU  Util   S    SA    AUU                                                                                         | 13%   13%   13%
Disk Util   FF                                                                                                     |  2%    4%    7%
Mem  Util   S     SU                                      UB        B                                              | 56%   56%   56%
Swap Util   U   UR          R                                                                                      | 17%   17%   17%
------------------------------------------------------------------------------------------------------------------------------------
                                                            PROCESS LIST                                                Users=   31
                              User      CPU Util     Cum     Disk           Thd
Process Name   PID   PPID Pri Name   (  200% max)    CPU   IO Rate    RSS   Cnt
--------------------------------------------------------------------------------
glance       11288   9399 154 USER1    0.4/ 1.1     0.6  0.0/ 0.8   7.2mb    1
midaemon      1411      1 -16 root      0.4/ 1.5 47129.6  0.0/ 0.0   8.6mb    3
syncer         368      1 154 root      0.2/ 0.3  9747.0  0.0/ 0.1   252kb    1
statdaemon       3      0 128 root      0.2/ 0.3  9396.6  0.0/ 0.0    32kb    1
onl_rts015    5543   5478 154 USER2      0.2/ 0.0     0.4  0.0/ 0.0   5.5mb    1
cmcld         1874   1870  20 root      0.2/ 0.3  9633.1  0.0/ 0.0   8.2mb    1
oninit        2404   2226 156 informix  0.0/ 0.0   100.5  0.0/ 0.0  26.5mb    1
oninit        2405   2226 156 informix  0.0/ 0.0    98.9  0.0/ 0.0  26.5mb    1
oninit        2406   2226 156 informix  0.0/ 0.0   100.7  0.0/ 0.0  26.5mb    1
oninit        2412   2226 156 informix  0.0/ 0.0   100.7  0.0/ 0.0  26.5mb    1
oninit        2410   2366 156 informix  0.0/ 0.0    96.5  0.0/ 0.0  26.3mb    1

"midaemon" is part of MeasureWare. Whether you need it running 24/7 depends on whether you are using MeasureWare to monitor your system performance in detail and whether you have enough disc space to hold the files it creates in /var/opt/perf/datafiles .
The on/off switch for "midaemon" is in /etc/rc.config.d/mwa .
If it is not running it will start itself for the duration of a "glance" session.

Using "glance" itself the following options are useful clues about performance:
m = memory report
t = system tables
B = global waits
i = i/o by filesystem

The figures you post don't show a system under great stress. The disc utilisation is so low that I wonder if the active portion of the database fits into its own buffers or whether you have very slow discs. Normally I would expect a database system to be disc-bound. I can't imagine that anyone would configure Informix to use raw disc nowadays but that might be an explanation.

Methyl would agree, we know not much of this system except it has 2 CPU and seems to run HP-UX 11.11...
I agree with methyl's last post and find the figures quite different from the initial post, I, like methyl would have appreciated the output of the command:

swapinfo -tam

Do you have batch running evening/night ?

---------- Post updated at 15:13 ---------- Previous update was at 15:07 ----------

What does:

grep Physi /var/adm/syslog/syslog.log

produce?

For comparison here is the output of my favorite box:

B3692A GlancePlus C.03.50.00    16:32:44      ant 9000/800                          Current  Avg  High
------------------------------------------------------------------------------------------------------
CPU  Util   S                        SN  NRU       U                                 | 57%   48%   77%
Disk Util   F          F                                                             | 16%   23%   71%
Mem  Util   S            SU             UB    B                                      | 49%   49%   49%
Networkil   U    UR      R                                                           | 19%   19%   19%
------------------------------------------------------------------------------------------------------
                                             PROCESS LIST                                 Users=   14

You say the CPU load is low but according to the CPU summary from the top output you have given us there is not much spare CPU capacity:

when you look at the idle columns...

I find running mpstat while a system is busy because it will show if your CPUs are I/O bound.

Adding a "top -b" into a log file in cron can be helpful to see what the resource hungry processes are at that time, where as all the other tools (sar, vmstat, mpstat) show you how busy the system is they don't show what processes are responsible!