I am an Oracle DBA with some strange IO performance problems.
Between 04:00 and 06:00 AM local time each day I see a degrading of
IO performance (to a SAN).
I see that the AIX diagnostics seem to begin at 04:00.
Is it likely that AIX diagnostics might be affecting IO?
Additional information:
Box is P55A 5.3 ML 7 or better (I am not root on this box).
None of the AIX monitors show any queuing, nor do the SAN
monitors show anything above 10ms response.
The database however shows, via statspack, average waits for
log file sync (writing to the redo logs) of over 500ms and numerous
timeouts (retrys). At other times of the day the average wait is more like 4ms or less with just a couple of timeouts.
The diag ela (ela = error log analysis) cronjob should only be checking the errpt / errlog file - which should be in rootvg - for certain types of errors, it should not be doing anything else - by default.
Ask the storage team to check if there is activity from other hosts on your back end storage at this time, such as backups, that might be affecting your disk access.
If your host is not busy and the wait seems to be on the san disks then either the san or the back end storage must be busy and causing the delay.
We have had the storage team checking everything they can, with no positive results.
These symptoms of degraded IO from 4-6AM just started after the box was rebooted.
I see the same symptoms on many AIX boxes to one degree or another during the same
time period. The degraded performance disappears after 6AM.
We have been trying to align the symptoms with some disk intensive process that happens somewhere, but no smoking gun yet. Will keep trying.
Another strange thing is I do not see the same symptoms from Sun Sparc Solaris boxes
connected to the same SAN during that same 4-6AM period of time, but those boxes
have fewer databases on them too.