Negative impact on Oracle DB after Solaris recommended patching

Dear All,

We are facing some performance problems in our oracle DB qerries and reports recently. We applied Solaris 10 recommended patchset on april 24th. Our monthly report generation acitivity ran 3 days i.e may 11,12,13 and application team complained of degraded performance. I can see that the cpu usage on the server is about 100% during the report generation , but everything else is fine. Free memory is 15gb , all mountpoints are under 80% and there is no other hardware issue as well.
Now app team is suspecting that last month patching could be the cause of the problem, because till last month they were performing this activity without having a performance degradtion.

I just need to be sure that solaris 10 recommended patches are not causing this issue. Any of you ever faced such an issue after patch installation ?
The server is V490 with 8 cpu / 32gb RAM. Any hints will be appreciated ?

Thanks in Advance.

You will have to either log a call to oracle support - be prepared with lots of logs, you will probably be asked to run explorer as well.

Or you can try searching by patch number on the OTN for known issues.

Has your DBA done a THROROUGH alert log check?

And, of course the DBA's have pushed no schema changes, there were zero production code changes.

If that fails startup sa (sar) sar monitoring to get a baseline. Keep 30 days worth of data. Then on month end start comparing daily sar output against what you get during month-end.

you may have something that reset your kernel parameters in /etc/system. I would look to ensure those have not changed as well

Hi os2mac,

nothing has changed in kernel parameters in /etc/system

Hi jim,
our DBA has logged call with Oracle and also shared the patches we applied last month to get a response from them. Lets wait and see what oracle has to say on this.

After checking logs, DBA could only confirm that there is very high Logical I/O activity. There is no wait I/O on the system and DBA also confirmed physical I/O is negligible from his logs.

Could it be that the high logical I/O is caused by some application defect ? Also we have lot of memory available on the system which is not used by oracle. Shall i ask DBA to increase SGA to include the free memory ? Will it help reduce the logical I/O. Any comments will be appreciated.

If you are running a modern version of the Oracle DB then you can use and AWR report to determine if the DB needs more ram allocated.

Tools like Toad and Spotlight are great for locating performance issues

Check with
vmstat 3
how CPU distributes on system and user, and with
prstat
how user CPU distributes on processes.

[LEFT]Hi MadeInGermany,

During the performance degradation, below was the status from top o/p

195 processes: 175 sleeping, 13 running, 1 zombie, 6 on cpu
CPU states: 0.0% idle, 78.4% user, 21.6% kernel, 0.0% iowait, 0.0% swap
Memory: 32G phys mem, 16G free mem, 70G total swap, 70G free swap

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
18964 oracle 39 41 0 6389M 5488M cpu/1 2:50 15.04% oracle
19638 oracle 26 59 0 6390M 5490M run 0:27 8.58% oracle
19648 oracle 36 31 0 6418M 5516M run 0:21 5.92% oracle
19023 oracle 34 30 0 6398M 5498M run 0:37 5.88% oracle
7229 oracle 38 30 0 6427M 5514M run 234:24 5.69% oracle
6912 oracle 45 31 0 6395M 5483M cpu/16 218:04 5.31% oracle
7216 oracle 28 31 0 6430M 5517M run 232:40 5.04% oracle
19644 oracle 35 40 0 6390M 5490M cpu/17 0:24 4.74% oracle
6897 oracle 112 30 0 6392M 5479M run 236:38 4.52% oracle
3813 oracle 106 31 0 6428M 5528M run 100:11 4.43% oracle
18970 oracle 234 31 0 6398M 5499M cpu/0 1:45 4.15% oracle
19749 oracle 1 32 0 6386M 5486M run 0:00 4.05% oracle
19646 oracle 23 31 0 6405M 5505M run 0:25 3.61% oracle
18972 oracle 11 32 0 6398M 5497M run 2:18 3.61% oracle
7220 oracle 258 32 0 6468M 5555M cpu/3 113:13 3.56% oracle

So approximately, there was an 80 - 20 ratio between user% and system% of cpu utilisation. Same thing i confirmed from vmstat as well.

Thanks in advance for your suggestions based on above log.[/LEFT]

---------- Post updated at 01:21 PM ---------- Previous update was at 12:45 PM ----------

Hi All,

In addition to this, i just wanted to know whether any of you have ever faced a performance problem in oracle DB after solaris patching ? I find this highly improbable, Just let me know your response on this .

Have you captured all the logs before you went for patching.

Like Shared memory / cache settings in /etc/system

Also.. Try to understand the DB/ Application (how it works) ,
How much GB does your DB needs..
What are the scheduled cron jobs.. Are the LUNS allocated to zpools or disk having any issues..

does your zpool having any permananent errors..

/var/adm/messages will help...

if the patches installed were os patches, there is a possibility that some corruption to the db has occured especially if the db was active during the patch install ... however -- whether or not db was active during patching -- if possible, restore a pre-patch copy of the db into another directory and see if that works better ...