Hi Folks,
I am facing an issue with the performance.
P4 with 1 processor and 16 GB RAM and SAN HDD = Oracle report takes 25 minutes
P5 with 2 processors and 16 GB RAM internall HDD with LPAR = Oracle Report takes 1 hour 15 minutes ( please note I have assigned all the max processors and memory to this LPAR and this is the only LPAR active )
Any help and insight would be appreciated
We have seen differences in performance of a similar scale and nature.
Internal/local disk is managed by your server and it is responsible for all the real IO, writing the bits to the disk controllers and getting the acknowledgement when the update is made to the brown-spinning part.
With SAN storage, the request by your OS to write data is accepted and confirmed by the SAN before the data is really written to disk. It's written to the SAN cache, but as far as your OS is concerned, then transaction is complete and processing can continue.
A SAN disk array will have a large cache dedicated to holding write requests and flushing them to real disk in large IO writes. This is why all SAN disk arrays will have batteries to keep the cache alive in the event of a power failure. This is in addition to any UPS you may have in place. A SAN shutdown can also be done in two ways roughly described as immediate and controlled. In the controlled shutdown, the cache is flushed and the SAN disk is consistent. An Immediate shutdown does not flush the cache and is quicker. It has no difference for the actual data stored unless the battery fails or is replaced.
As to a way to measure it, I suppose you are into creating a large complex file and copying it about a few times. By complex, I mean something that compression software will not easily deal with. My way to do this (mainly for testing FTP speed) is to tar up some large files and compress them, then append them a few times to get the size big enough.
If your hardware can support it, then it can be beneficial to have everything as SAN storage, including the OS and paging space. The limitation then becomes the connection speed to the SAN.
If you replicate your SAN to another site, then this takes care of updates to the boot disks / rootvg / vg00 (whatever you want to call it) that may affect the DR position. Of course, the IP address will be replicated so you have to handle that along with a process to use perhaps different hardware at the remote site.
It is, however no substitute for backups as errors / corruption will also be replicated honestly.
I hope that this helps.
Robin
Liverpool/Blackburn
UK
This type of question really need one to drill down to the core of the system and perform an autopsy.
Is both OS and DB version same and size of DB equal on those systems?
Are you running same queries (doing the same activity as on the other system)?
If no, then what is the version of OS on both? Have you tuned the system on latter as per the standard? (like Memory tuning).
How ample is the network bandwidth, what is the type of storage.
What version of oracle DB you are using on both?
Are there any patches for oracle and APAR for AIX to apply and have you looked out for those?
In short these types of questions cannot be answered easily.
Just to add to ibmtechs list (which is very good already):
What are the respective sizes of the SGA?
What is the output of "vmstat -v"?
I hope this helps.
bakunin
Okay,
#1 Rbattle
I am using P550 VIOS so it cannot have a dedicated SAN HBA card,
Have to first assign the HDISK from SAN to I/O VIOS server and then from VIOS to LPAR
#2 IBMTECH and Bakunin
Yes, both database size are exactly same, infact it is a clone of the production environment, the min max paging space and other parameters are also same.... everything is same. DATA, DATABASE, Parameters, OS,
only difference
ORADB = Production Environment runs on P4 and Oracle is on SAN = report take 25 minutes
CloneDB = Clone Environment runs on P5 and Oracle is on internal Disks = report takes 1 hour and 15 minutes
AIX P4 = PRODUCTION
root@oradb:/>pagesize -a
4096
root@oradb:/>vmstat -v
4194304 memory pages
4000247 lruable pages
9961 free pages
1 memory pools
218074 pinned pages
80,1 maxpin percentage
20,0 minperm percentage
80,0 maxperm percentage
80,3 numperm percentage
3214895 file pages
0,0 compressed percentage
0 compressed pages
0,0 numclient percentage
80,0 maxclient percentage
0 client pages
0 remote pageouts scheduled
53 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
80406 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
AIX P5 ON LPAR = CLONE
pagesize -a
4096
root@clodb:/>vmstat -v
2097152 memory pages
2017797 lruable pages
1713122 free pages
2 memory pools
121734 pinned pages
80.0 maxpin percentage
5.0 minperm percentage
90.0 maxperm percentage
0.7 numperm percentage
14208 file pages
0.0 compressed percentage
0 compressed pages
1.2 numclient percentage
90.0 maxclient percentage
24760 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
2740 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
ORACLE ON CLONE AIX ENV P5 LPAR
oraprod@clodb:/oratech/proddb/9.2.0/>./startup.sh
You are running addbctl.sh version 115.8
Starting the database PROD ...
SQL*Plus: Release 9.2.0.3.0 - Production on Wed Jan 15 14:16:00 2014
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Connected to an idle instance.
ORACLE instance started.
Total System Global Area 615482512 bytes
Fixed Size 742544 bytes
Variable Size 436207616 bytes
Database Buffers 167772160 bytes
Redo Buffers 10760192 bytes
Database mounted.
Database opened.
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.3.0 - Production
addbctl.sh: exiting with status 0
You are running addlnctl.sh version 115.6
Logfile: /oratech/proddb/9.2.0/appsutil/log/PROD_clodb/addlnctl.txt
Starting listener process PROD ...
addlnctl.sh: exiting with status 0
oraprod@clodb:/oratech/proddb/9.2.0/>
oraprod@clodb:/oratech/proddb/9.2.0/>sqlplus
SQL*Plus: Release 9.2.0.3.0 - Production on Wed Jan 15 14:19:04 2014
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Enter user-name: sys
Enter password:
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.3.0 - Production
SQL> connect as sysdba
Enter user-name: sys
Enter password:
Connected.
SQL> show parameters area_size
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
bitmap_merge_area_size integer 1048576
create_bitmap_area_size integer 8388608
hash_area_size integer 131072
sort_area_size integer 65536
workarea_size_policy string AUTO
SQL> set pages 999; column pga_size format 999,999,999;
SQL> select
1048576+a.value+b.value pga_size
from
v$parameter a,
v$parameter b
where
a.name = 'sort_area_size'
and
b.name = 'hash_area_size'
;
PGA_SIZE
------------
1,245,184
This way it sounds more logical, because SAN disks are (in most cases) much faster than physical disks. They have usually a lot more IOPS and more throughput bandwidth, because a lot of different reasons: stripesets, caching controllers, buses with higher bandwidth, ....
Your systems seem not to have the same amount of memory at all:
root@oradb:/>vmstat -v
4194304 memory pages
[...]
root@clodb:/>vmstat -v
2097152 memory pages
[...]
Further this looks dubious:
root@oradb:/>vmstat -v
[...]
80406 filesystem I/Os blocked with no fsbuf
[...]
whereas:
root@clodb:/>vmstat -v
[...]
2740 filesystem I/Os blocked with no fsbuf
[...]
Not the value in itself is problematic, but the huge difference. Monitor the value closely over time, if it increases heavily you have found a potential bottleneck. If it stays at this level it is perhaps an artefact of some temporary memory shortness.
Finally, the tuning parameters seem to be different (compare "maxperm" and "minperm" in the different outputs, probably others are different too). You should run "vmo -a" (and the other tuning utilities, "schedo", "ioo", "no") on both machines to investigate other differences. What good values for maxperm and minperm would be is hard to suggest because it depends heavily on the (detailed) OS version which we do not know.
Another supposition (which would have to be proven by facts) of mine is that the faster processor is not needed in the special kind of report you run and therefore contributes nothing to some faster execution, whereas the disks contribute heavily.
I hope this helps.
bakunin
And to add more to bakunin's finding, have you enable AIO? ( I assume you are using AIX5.3, given the specification of hardware). Did you cross verify that CIO option for DB file systems is enabled on Prod system (if AIX5.3 or less).
Can you provide us the VMM parameters that are currently configured in both systems?
Other parameters like
all have been enabled and oracle documentation has been followed
lsattr -El aio0
autoconfig available STATE to be configured at system restart True
fastpath enable State of fast path True
kprocprio 39 Server PRIORITY True
maxreqs 4096 Maximum number of REQUESTS True
maxservers 10 MAXIMUM number of servers per cpu True
minservers 1 MINIMUM number of servers True
root@clodb:/>ps -kf | grep aio
root 86172 1 0 14:00:14 - 0:00 posix_aioserver
root 90168 1 0 14:00:14 - 0:00 aioserver
root 135390 1 0 14:08:42 - 0:00 aioserver
root 282784 1 0 14:08:45 - 0:00 aioserver
root 319648 1 0 14:08:42 - 0:00 aioserver
root 323742 1 0 14:08:42 - 0:00 aioserver
root 331944 1 0 14:08:45 - 0:00 aioserver
root 336036 1 0 14:08:46 - 0:00 aioserver
root 340134 1 0 14:08:46 - 0:00 aioserver
root 344232 1 0 14:08:46 - 0:00 aioserver
root 348330 1 0 14:08:46 - 0:00 aioserver
root 352428 1 0 14:08:47 - 0:00 aioserver
root 356526 1 0 14:08:47 - 0:00 aioserver
root 360624 1 0 14:08:47 - 0:00 aioserver
root 364722 1 0 14:08:47 - 0:00 aioserver
root 368820 1 0 14:08:47 - 0:00 aioserver
root 372918 1 0 14:08:47 - 0:00 aioserver
root 377016 1 0 14:08:47 - 0:00 aioserver
root 381114 1 0 14:08:47 - 0:00 aioserver
root 385212 1 0 14:08:47 - 0:00 aioserver
root 389310 1 0 14:08:48 - 0:00 aioserver
root@clodb:/>iostat -A
System configuration: lcpu=2 drives=3 ent=0.10
aio: avgc avfc maxg maif maxr avg-cpu: % user % sys % idle % iowait physc % entc
0 0 945 0 4096 8.8 0.1 36.8 54.4 0.0 9.2
" Disk history since boot not available. "
in the above you can see maxreqs= 4096
root@clodb:/>lsattr -El aio0 -a maxreqs
maxreqs 4096 Maximum number of REQUESTS True
Before running the report:
root@clodb:/>vmstat -v
2097152 memory pages
2017797 lruable pages
1704079 free pages
2 memory pools
117776 pinned pages
80.0 maxpin percentage
5.0 minperm percentage
90.0 maxperm percentage
0.4 numperm percentage
9366 file pages
0.0 compressed percentage
0 compressed pages
0.9 numclient percentage
90.0 maxclient percentage
18919 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
2740 filesystem I/Os blocked with no fsbuf
10349 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
After running the report Parts Potential Report - Order Management
root@clodb:/>vmstat -v
2097152 memory pages
2017797 lruable pages
1695656 free pages
2 memory pools
118544 pinned pages
80.0 maxpin percentage
5.0 minperm percentage
90.0 maxperm percentage
0.5 numperm percentage
11262 file pages
0.0 compressed percentage
0 compressed pages
1.0 numclient percentage
90.0 maxclient percentage
21501 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
2740 filesystem I/Os blocked with no fsbuf
10349 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
root@clodb:/>
root@clodb:/>svmon -G
size inuse free pin virtual
memory 2097152 401985 1695167 118642 380315
pg space 131072 2627
work pers clnt lpage
pin 118642 0 0 0
in use 380315 0 21670 0
After running the report topas shows
Topas Monitor for host: clodb EVENTS/QUEUES FILE/TTY
Mon Jan 27 16:05:45 2014 Interval: 2 Cswitch 649 Readch 1740.2K
Syscall 2005 Writech 14514
Kernel 1.0 |# | Reads 241 Rawin 0
User 94.5 |########################### | Writes 30 Ttyout 122
Wait 4.5 |## | Forks 0 Igets 0
Idle 0.0 | | Execs 0 Namei 0
Physc = 1.00 %Entc= 999.3 Runqueue 1.0 Dirblk 0
Waitqueue 0.0
Network KBPS I-Pack O-Pack KB-In KB-Out
en0 34.5 155.0 182.0 43.9 25.9 PAGING MEMORY
lo0 3.4 84.0 84.0 3.5 3.5 Faults 3 Real,MB 8191
Steals 0 % Comp 19.1
Disk Busy% KBPS TPS KB-Read KB-Writ PgspIn 0 % Noncomp 1.0
hdisk7 100.0 1272.0 169.0 1496.0 1048.0 PgspOut 0 % Client 1.5
hdisk6 0.0 0.0 0.0 0.0 0.0 PageIn 0
hdisk8 0.0 0.0 0.0 0.0 0.0 PageOut 0 PAGING SPACE
Sios 0 Size,MB 512
Name PID CPU% PgSp Owner % Used 2.5
rmcd 123114 50.0 10.0 root NFS (calls/sec) % Free 97.4
oracle 680012 0.0 5.2 oraprod ServerV2 0
dtterm 671820 0.0 1.3 root ClientV2 0 Press:
topas 520302 0.0 1.6 root ServerV3 0 "h" for help
oracle 495862 0.0 4.3 oraprod ClientV3 0 "q" to quit
VMO VMM Parameters optimized to the max
root@clodb:/>vmo -a
cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = 4096
large_page_heap_size = 0
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 0
lru_poll_interval = 0
lrubucket = 131072
maxclient% = 90
maxfree = 1088
maxperm = 1816016
maxperm% = 90
maxpin = 1677722
maxpin% = 80
mbuf_heap_psize = 4096
memory_affinity = 1
memory_frames = 2097152
memplace_data = 2
memplace_mapped_file = 2
memplace_shm_anonymous = 2
memplace_shm_named = 2
memplace_stack = 2
memplace_text = 2
memplace_unmapped_file = 2
mempools = 1
minfree = 960
minperm = 100889
minperm% = 5
nokilluid = 0
npskill = 1024
npsrpgmax = 8192
npsrpgmin = 6144
npsscrubmax = 8192
npsscrubmin = 6144
npswarn = 4096
num_spec_dataseg = 0
numpsblks = 131072
page_steal_method = 1
pagecoloring = n/a
pinnable_frames = 1978524
pta_balance_threshold = n/a
relalias_percentage = 0
rpgclean = 0
rpgcontrol = 2
scrub = 0
scrubclean = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = 512
strict_maxclient = 1
strict_maxperm = 0
v_pinshm = 0
vm_modlist_threshold = -1
vmm_fork_policy = 1
root@clodb:/>
root@clodb:/>vmo -L
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
cpu_scale_memp 8 8 8 1 64 B
--------------------------------------------------------------------------------
data_stagger_interval 161 161 161 0 0 4KB pages D
lgpg_regions
--------------------------------------------------------------------------------
defps 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
framesets 2 2 2 1 10 B
--------------------------------------------------------------------------------
htabscale n/a -1 -1 -4 0 B
--------------------------------------------------------------------------------
kernel_heap_psize 4K 4K 4K 4K 16M bytes B
lgpg_regions
--------------------------------------------------------------------------------
large_page_heap_size 0 0 0 0 8E-1 bytes B
lgpg_regions
--------------------------------------------------------------------------------
lgpg_regions 0 0 0 0 D
lgpg_size
--------------------------------------------------------------------------------
lgpg_size 0 0 0 0 16M bytes D
lgpg_regions
--------------------------------------------------------------------------------
low_ps_handling 1 1 1 1 2 D
--------------------------------------------------------------------------------
lru_file_repage 0 1 0 0 1 boolean D
--------------------------------------------------------------------------------
lru_poll_interval 0 0 0 0 60000 milliseconds D
--------------------------------------------------------------------------------
lrubucket 128K 128K 128K 64K 4KB pages D
--------------------------------------------------------------------------------
maxclient% 90 80 90 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
maxfree 1088 1088 1088 8 200K 4KB pages D
minfree
memory_frames
--------------------------------------------------------------------------------
maxperm 1773K 1773K S
--------------------------------------------------------------------------------
maxperm% 90 80 90 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
maxpin 1638K 1638K S
--------------------------------------------------------------------------------
maxpin% 80 80 80 1 99 % memory D
pinnable_frames
memory_frames
--------------------------------------------------------------------------------
mbuf_heap_psize 4K 4K 4K 4K 16M bytes B
--------------------------------------------------------------------------------
memory_affinity 1 1 boolean d
memplace_text
memplace_data
memplace_stack
memplace_shm_named
memplace_shm_anonymous
memplace_mapped_file
memplace_unmapped_file
--------------------------------------------------------------------------------
memory_frames 2M 2M 4KB pages S
--------------------------------------------------------------------------------
memplace_data 2 2 2 1 2 D
--------------------------------------------------------------------------------
memplace_mapped_file 2 2 2 1 2 D
--------------------------------------------------------------------------------
memplace_shm_anonymous 2 2 2 1 2 D
--------------------------------------------------------------------------------
memplace_shm_named 2 2 2 1 2 D
--------------------------------------------------------------------------------
memplace_stack 2 2 2 1 2 D
--------------------------------------------------------------------------------
memplace_text 2 2 2 1 2 D
--------------------------------------------------------------------------------
memplace_unmapped_file 2 2 2 1 2 D
--------------------------------------------------------------------------------
mempools 1 1 1 1 2 B
--------------------------------------------------------------------------------
minfree 960 960 960 8 200K 4KB pages D
maxfree
memory_frames
--------------------------------------------------------------------------------
minperm 100889 100889 S
--------------------------------------------------------------------------------
minperm% 5 20 5 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
nokilluid 0 0 0 0 4G-1 uid D
--------------------------------------------------------------------------------
npskill 1K 1K 1K 1 128K-1 4KB pages D
--------------------------------------------------------------------------------
npsrpgmax 8K 8K 8K 0 128K-1 4KB pages D
npsrpgmin
--------------------------------------------------------------------------------
npsrpgmin 6K 6K 6K 0 128K-1 4KB pages D
npsrpgmax
--------------------------------------------------------------------------------
npsscrubmax 8K 8K 8K 0 128K-1 4KB pages D
npsscrubmin
--------------------------------------------------------------------------------
npsscrubmin 6K 6K 6K 0 128K-1 4KB pages D
npsscrubmax
--------------------------------------------------------------------------------
npswarn 4K 4K 4K 0 128K-1 4KB pages D
--------------------------------------------------------------------------------
num_spec_dataseg 0 0 0 0 B
--------------------------------------------------------------------------------
numpsblks 128K 128K 4KB pages S
--------------------------------------------------------------------------------
page_steal_method 1 1 1 0 1 boolean B
--------------------------------------------------------------------------------
pagecoloring n/a 0 0 0 1 boolean B
--------------------------------------------------------------------------------
pinnable_frames 1932K 1932K 4KB pages S
--------------------------------------------------------------------------------
pta_balance_threshold n/a 1 1 0 99 % pta segment D
--------------------------------------------------------------------------------
relalias_percentage 0 0 0 0 32K-1 D
--------------------------------------------------------------------------------
rpgclean 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
rpgcontrol 2 2 2 0 3 D
--------------------------------------------------------------------------------
scrub 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
scrubclean 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
soft_min_lgpgs_vmpool 0 0 0 0 90 % D
lgpg_regions
--------------------------------------------------------------------------------
spec_dataseg_int 512 512 512 0 B
--------------------------------------------------------------------------------
strict_maxclient 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
strict_maxperm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
v_pinshm 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
vm_modlist_threshold -1 -1 -1 -2 2G-1 D
--------------------------------------------------------------------------------
vmm_fork_policy 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
n/a means parameter not supported by the current platform or kernel
Parameter types:
S = Static: cannot be changed
D = Dynamic: can be freely changed
B = Bosboot: can only be changed using bosboot and reboot
R = Reboot: can only be changed during reboot
C = Connect: changes are only effective for future socket connections
M = Mount: changes are only effective for future mountings
I = Incremental: can only be incremented
d = deprecated: deprecated and cannot be changed
Value conventions:
K = Kilo: 2^10 G = Giga: 2^30 P = Peta: 2^50
M = Mega: 2^20 T = Tera: 2^40 E = Exa: 2^60
root@clodb:/>
root@clodb:/>vmo -L minperm% -L maxperm% -L maxclient% -L lru_file_repage
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
lru_file_repage 0 1 0 0 1 boolean D
--------------------------------------------------------------------------------
maxclient% 90 80 90 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
maxperm% 90 80 90 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
minperm% 5 20 5 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
root@clodb:/>
PRODUCTION MACHINE PARAMETERS
root@oradb:/>vmo -a
memory_frames = 4194304
pinnable_frames = 3976053
maxfree = 128
minfree = 120
minperm% = 20
minperm = 800049
maxperm% = 80
maxperm = 3200197
strict_maxperm = 0
maxpin% = 80
maxpin = 3355444
maxclient% = 80
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 2228224
npskill = 17408
npswarn = 69632
v_pinshm = 0
pta_balance_threshold = 50
pagecoloring = 0
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = n/a
spec_dataseg_int = n/a
memory_affinity = 1
htabscale = -1
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = n/a
kernel_heap_psize = n/a
soft_min_lgpgs_vmpool = 0
vmm_fork_policy = 0
low_ps_handling = 1
mbuf_heap_psize = n/a
strict_maxclient = 1
cpu_scale_memp = 8
lru_poll_interval = 0
lru_file_repage = 1
root@oradb:/>
root@oradb:/>vmo -L minperm% -L maxperm% -L maxclient% -L lru_file_repage
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
minperm% 20 20 20 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
maxperm% 80 80 80 1 100 % memory D
minperm%
maxclient%
--------------------------------------------------------------------------------
maxclient% 80 80 80 1 100 % memory D
maxperm%
--------------------------------------------------------------------------------
lru_file_repage 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
root@oradb:/>
My only suspect here is SAN Disk vs Internal Disk, this is what I could zoom into. Anyone think the problem could be other than HDD , please let me know.