I have attached the nmon report extracted from one of my systems.
I've never seen something like this. It appears that all the cpu usage is being utilized on one CPU and the others appear to be idle or hardly used. I've seen this in a few systems recently. Now I know that their paging space is not setup correctly. and I'm going though the steps to show them that it needs to be changed. I have attached a simplified nmon log if someone can look at it and tell me if they can see why only one cpu is being utilized would be great as I've not seen this before.
Here is the lparstat
Partition Number : 4
Type : Shared-SMT-4
Mode : Capped
Entitled Capacity : 4.00
Partition Group-ID : 32772
Shared Pool ID : 0
Online Virtual CPUs : 40
Maximum Virtual CPUs : 60
Minimum Virtual CPUs : 1
Online Memory : 65536 MB
Maximum Memory : 131072 MB
Minimum Memory : 4096 MB
Variable Capacity Weight : 0
Minimum Capacity : 0.10
Maximum Capacity : 6.00
Capacity Increment : 0.01
Maximum Physical CPUs in system : 16
Active Physical CPUs in system : 16
Active CPUs in Pool : 16
Shared Physical CPUs in system : 16
Maximum Capacity of Pool : 1600
Entitled Capacity of Pool : 1500
Unallocated Capacity : 0.00
Physical CPU Percentage : 10.00%
Unallocated Weight : 0
Memory Mode : Dedicated
Total I/O Memory Entitlement : -
Variable Memory Capacity Weight : -
Memory Pool ID : -
Physical Memory in the Pool : -
Hypervisor Page Size : -
Unallocated Variable Memory Capacity Weight: -
Unallocated I/O Memory entitlement : -
Memory Group ID of LPAR : -
Desired Virtual CPUs : 40
Desired Memory : 65536 MB
Desired Variable Capacity Weight : 0
Desired Capacity : 4.00
Target Memory Expansion Factor : -
Target Memory Expansion Size : -
Power Saving Mode : Disabled
Nobody needs so much paging space. Remove paging01 and 00 and shring hd6 to something reasonable like maybe 10GB to be on the safe side. If your system ever uses paging space, it will start to be very slow, usually. If the disks are SAN disks, it could be that it might a tad faster, but I would try to have enough memory or a tuned box to avoid paging at all costs. 54GB of paging space makes no sense at all.
If anything continously uses paging space, you most probably have a memory leak somewhere. You can check processes using paging space with svmon -P ; maybe consult the man page for details.
I opened the nmon excel sheet, but I only see some overall statistics and something about CPU 4. You can use something like sar -P ALL 1 999 for example to have check for how much load is on your logical CPUs, but since there is SMT activated and the logical CPUs are no real CPUs, you can't say for sure which CPU is having much taffic or not. I was curious myself some time ago and asked Nigel Griffiths (nagger), author of nmon and a well-known AIX engineer/architect(?) in this thread (me as player2): developerWorks : AIX and UNIX : Performance Tools Forum : NMON: CPU total graph does not match ...
What might be interessting is how the CPUs dispatch parallel requests. You can check this with mpstat -s 1 1 for utilization of your logical CPUs. If they are not about close percentage per non logical CPU, it might not be of a high benefit to have SMT activated. Can try this out maybe. You can turn off etc. SMT with smtctl ; check the man page for it.
You can also check if your processes are using multithreading, ie. take advantage of SMT by using svmon -P and check for the column with the appropriate heading "mthread" or something, don't remember exactly.
Like zaxxon I agree that you have excessive paging space. And, basically, the fact that your application is using paging space in spite of having 64G byte of activated memory implies a poorly configured application. "Paging space" is so 1970-80's.
I have not looked at your nmon data, but from what you say in your first post it sounds as if your application is mono-threaded. This can be verified yes/no using
$ ps -emo THREAD
There are several ways to get an idea of how your processors are being scheduled: sar, topas, nmon (actually topas and nmon are the same program, different views of data), mpstat. However, I do not recommend 1 sec intervals. 5 sec is what I use when I am in a hurry, 10/15/60 seconds is what I prefer when observing a system live.
For a 24 hour nmon collection (to be viewed using nmon analyzer) I use a 360 second interval (6 minutes) for 240 samples per day.
Most (or all) Unix'es and Linux'es show a very high mem usage as the part of RAM, that is not needed for processes, is used for file caching. Nothing to worry about, that's just normal.
concerning the load average. I should be concerned about this now as that seems a bit high right? during the test i didn't see it reach as high but close to.
Yes, the load seems alarming.
nmon should show you, which processes/application are the cause of this. If not on day time, setup nmon for continuous logging.
You have 40 Online Virtual CPUs with SMT 4x enabled =
160 logical CPUs but only 4 phyical CPUs (Entitled Capacity)
I would suggest to reduce the amount of virtual CPUs (start with 4) and if this is not enough try to increase the amount of virtual CPUs and Entitled Capacity.
If Your processes are not multi threaded aware or they are not enough processes which can be scheduled, it should be helpful to disable SMT.
The paging space # of 2.4% does not really mean much, because you have too much paging space to begin with. You can just turn them off the two extra ones with the swapoff command (and perhaps later rmps to remove them permanently).
vpundit: I was not aware that the rest of UNIX had caught up with AIX - and using memory for file cacheing - good to hear.
regarding the ratio of entitled capacity to virtual processors: your ratio of 1:10, while legal is not considered best practice. as mentioned "xRay" paying attention to SRAD information is important.
use the mpstat command to see if you are using a single thread on each processor - if so, count the actual processors being activated, divide that number in two and use that as your number of VP - that should lower your physical processor cost.
lparstat output is important to see "app" value - as an estimate of how busy the system as whole is busy.
but in all honesty, there is so much that needs to be done, it is all patchwork until some proper settings are applied, and all testing restarted.