# Not using all CPU's? **URL:** \ **Category:** AIX\ **Created:** [November 22, 2012, 11:33am UTC](https://community.unix.com/t/not-using-all-cpus/320231 "2012-11-22T11:33:40Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![vpundit](https://community.unix.com/letter_avatar/vpundit/32/5_5575768a8748004e209b776fc1b2916d.png) [@vpundit](https://community.unix.com/u/vpundit)\ **Post date:** [November 22, 2012, 11:33am UTC](https://community.unix.com/t/not-using-all-cpus/320231/1 "2012-11-22T11:33:40Z")
Hi guys, I am a bit confused here, I have attached the nmon report extracted from one of my systems. I've never seen something like this. It appears that all the cpu usage is being utilized on one CPU and the others appear to be idle or hardly used. I've seen this in a few systems recently. Now I know that their paging space is not setup correctly. and I'm going though the steps to show them that it needs to be changed. I have attached a simplified nmon log if someone can look at it and tell me if they can see why only one cpu is being utilized would be great as I've not seen this before. Here is the lparstat ```nohighlight Partition Number : 4 Type : Shared-SMT-4 Mode : Capped Entitled Capacity : 4.00 Partition Group-ID : 32772 Shared Pool ID : 0 Online Virtual CPUs : 40 Maximum Virtual CPUs : 60 Minimum Virtual CPUs : 1 Online Memory : 65536 MB Maximum Memory : 131072 MB Minimum Memory : 4096 MB Variable Capacity Weight : 0 Minimum Capacity : 0.10 Maximum Capacity : 6.00 Capacity Increment : 0.01 Maximum Physical CPUs in system : 16 Active Physical CPUs in system : 16 Active CPUs in Pool : 16 Shared Physical CPUs in system : 16 Maximum Capacity of Pool : 1600 Entitled Capacity of Pool : 1500 Unallocated Capacity : 0.00 Physical CPU Percentage : 10.00% Unallocated Weight : 0 Memory Mode : Dedicated Total I/O Memory Entitlement : - Variable Memory Capacity Weight : - Memory Pool ID : - Physical Memory in the Pool : - Hypervisor Page Size : - Unallocated Variable Memory Capacity Weight: - Unallocated I/O Memory entitlement : - Memory Group ID of LPAR : - Desired Virtual CPUs : 40 Desired Memory : 65536 MB Desired Variable Capacity Weight : 0 Desired Capacity : 4.00 Target Memory Expansion Factor : - Target Memory Expansion Size : - Power Saving Mode : Disabled ``` During that day here was the nmon uptime: ```nohighlight 12:04AM up 2 days, 9 hrs, 0 users, load average: 39.55, 42.90, 44.14 ``` ---
**Author:** ![vpundit](https://community.unix.com/letter_avatar/vpundit/32/5_5575768a8748004e209b776fc1b2916d.png) [@vpundit](https://community.unix.com/u/vpundit)\ **Post date:** [November 22, 2012, 11:45am UTC](https://community.unix.com/t/not-using-all-cpus/320231/2 "2012-11-22T11:45:15Z")
also here is the paging: ```nohighlight Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum paging01 hdisk715 rootvg 12800MB 4 yes yes lv 0 paging00 hdisk715 rootvg 12800MB 4 yes yes lv 0 hd6 hdisk714 rootvg 29952MB 2 yes yes lv 0 ``` ---
**Author:** ![zaxxon](https://community.unix.com/user_avatar/community.unix.com/zaxxon/32/660_2.png) [@zaxxon](https://community.unix.com/u/zaxxon)\ **Post date:** [November 22, 2012, 1:19pm UTC](https://community.unix.com/t/not-using-all-cpus/320231/3 "2012-11-22T13:19:20Z")
1. Nobody needs so much paging space. Remove paging01 and 00 and shring hd6 to something reasonable like maybe 10GB to be on the safe side. If your system ever uses paging space, it will start to be very slow, usually. If the disks are SAN disks, it could be that it might a tad faster, but I would try to have enough memory or a tuned box to avoid paging at all costs. 54GB of paging space makes no sense at all. If anything continously uses paging space, you most probably have a memory leak somewhere. You can check processes using paging space with `svmon -P` ; maybe consult the man page for details. 2. I opened the nmon excel sheet, but I only see some overall statistics and something about CPU 4. You can use something like `sar -P ALL 1 999` for example to have check for how much load is on your logical CPUs, but since there is SMT activated and the logical CPUs are no real CPUs, you can't say for sure which CPU is having much taffic or not. I was curious myself some time ago and asked Nigel Griffiths (nagger), author of nmon and a well-known AIX engineer/architect(?) in this thread (me as player2): [developerWorks : AIX and UNIX : Performance Tools Forum : NMON: CPU total graph does not match ...](http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14558287) What might be interessting is how the CPUs dispatch parallel requests. You can check this with `mpstat -s 1 1` for utilization of your logical CPUs. If they are not about close percentage per non logical CPU, it might not be of a high benefit to have SMT activated. Can try this out maybe. You can turn off etc. SMT with `smtctl` ; check the man page for it. You can also check if your processes are using multithreading, ie. take advantage of SMT by using `svmon -P` and check for the column with the appropriate heading "mthread" or something, don't remember exactly. ---
**Author:** ![MichaelFelt](https://community.unix.com/letter_avatar/michaelfelt/32/5_5575768a8748004e209b776fc1b2916d.png) [@MichaelFelt](https://community.unix.com/u/MichaelFelt)\ **Post date:** [November 22, 2012, 2:06pm UTC](https://community.unix.com/t/not-using-all-cpus/320231/4 "2012-11-22T14:06:03Z")
Like zaxxon I agree that you have excessive paging space. And, basically, the fact that your application is using paging space in spite of having 64G byte of activated memory implies a poorly configured application. "Paging space" is so 1970-80's. I have not looked at your nmon data, but from what you say in your first post it sounds as if your application is mono-threaded. This can be verified yes/no using $ ps -emo THREAD There are several ways to get an idea of how your processors are being scheduled: sar, topas, nmon (actually topas and nmon are the same program, different views of data), mpstat. However, I do not recommend 1 sec intervals. 5 sec is what I use when I am in a hurry, 10/15/60 seconds is what I prefer when observing a system live. For a 24 hour nmon collection (to be viewed using nmon analyzer) I use a 360 second interval (6 minutes) for 240 samples per day. ---
**Author:** ![vpundit](https://community.unix.com/letter_avatar/vpundit/32/5_5575768a8748004e209b776fc1b2916d.png) [@vpundit](https://community.unix.com/u/vpundit)\ **Post date:** [November 22, 2012, 2:48pm UTC](https://community.unix.com/t/not-using-all-cpus/320231/5 "2012-11-22T14:48:04Z")
thanks for the info guys! really appriciate it. I know the paging space is excessive i haven't been here to long, i keep finding things like this. I have to try some of those commands and run some compaires. I had a test running just a few min ago and memory is hitting 99%. ```nohighlight Memory Physical PageSpace | pages/sec In Out | FileSystemCache % Used 99.5% 2.4% | to Paging Space 0.0 0.0 | (numperm) 14.0% % Free 0.5% 97.6% | to File System 12.5 3443.5 | Process 77.7% MB Used 65224.7MB 1311.1MB | Page Scans 0.0 | System 7.8% MB Free 311.3MB 54240.9MB | Page Cycles 0.0 | Free 0.5% Total(MB) 65536.0MB 55552.0MB | Page Steals 0.0 | ----- | Page Faults 4297.0 | Total 100.0% ``` ---
**Author:** ![zaxxon](https://community.unix.com/user_avatar/community.unix.com/zaxxon/32/660_2.png) [@zaxxon](https://community.unix.com/u/zaxxon)\ **Post date:** [November 22, 2012, 3:01pm UTC](https://community.unix.com/t/not-using-all-cpus/320231/6 "2012-11-22T15:01:06Z")
Most (or all) Unix'es and Linux'es show a very high mem usage as the part of RAM, that is not needed for processes, is used for file caching. Nothing to worry about, that's just normal. ---
**Author:** ![vpundit](https://community.unix.com/letter_avatar/vpundit/32/5_5575768a8748004e209b776fc1b2916d.png) [@vpundit](https://community.unix.com/u/vpundit)\ **Post date:** [November 22, 2012, 3:14pm UTC](https://community.unix.com/t/not-using-all-cpus/320231/7 "2012-11-22T15:14:48Z")
> [@zaxxon](#): > > Most (or all) Unix'es and Linux'es show a very high mem usage as the part of RAM, that is not needed for processes, is used for file caching. Nothing to worry about, that's just normal. thank you! just one more question, concerning the load average. I should be concerned about this now as that seems a bit high right? during the test i didn't see it reach as high but close to. ---
**Author:** ![zaxxon](https://community.unix.com/user_avatar/community.unix.com/zaxxon/32/660_2.png) [@zaxxon](https://community.unix.com/u/zaxxon)\ **Post date:** [November 22, 2012, 3:23pm UTC](https://community.unix.com/t/not-using-all-cpus/320231/8 "2012-11-22T15:23:14Z")
Yes, the load seems alarming. nmon should show you, which processes/application are the cause of this. If not on day time, setup nmon for continuous logging. ---
**Author:** ![\_XrAy](https://community.unix.com/user_avatar/community.unix.com/_xray/32/2744_2.png) [@\_XrAy](https://community.unix.com/u/_XrAy)\ **Post date:** [November 23, 2012, 7:24am UTC](https://community.unix.com/t/not-using-all-cpus/320231/9 "2012-11-23T07:24:30Z")
> [@vpundit](#): > > LPAR is not using all CPU's For security reasons I can't review the nmon-file. It would be helpful when you post the output from ```plaintext mpstat -wa ``` ```plaintext lparstat -H ``` ```plaintext lssrad -av ``` In my opinion Your LPAR-Design isn't really good. There are some realy good articels from Nigel Griffith which i recommend to read: [https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/mpstat\_d\_and\_the\_undocumented\_stats133?lang=de](https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/mpstat%5C_d%5C_and%5C_the%5C_undocumented_stats133?lang=de) and please search for "local near far memory part " on google. [Local, Nar & Far Memory part 5 - Low Entitlement has a Bad Side - IBM](http://www.google.de/url?url=https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/local_nar_far_memory_part_5_low_entitlement_has_a_bad_side_effect4&rct=j&sa=U&ei=vm6vUNTuAYn34QTC-oDwBQ&ved=0CCEQFjAF&q=local+near++far+memory+part+7&usg=AFQjCNETAYlmGkw5uBqeJzd901d4KglA6Q) [Local, Nar & Far Memory part 6 - Too High a Virtual Processor - IBM](http://www.google.de/url?url=https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/local_nar_far_memory_part_6_too_high_a_virtual_processor_number_has_a_bad_side_effect198&rct=j&sa=U&ei=vm6vUNTuAYn34QTC-oDwBQ&ved=0CCMQFjAG&q=local+near++far+memory+part+7&usg=AFQjCNFXalWfbR8bkeNOfDh8ss5TIAObvw) You have 40 Online Virtual CPUs with SMT 4x enabled = 160 logical CPUs but only 4 phyical CPUs (Entitled Capacity) I would suggest to reduce the amount of virtual CPUs (start with 4) and if this is not enough try to increase the amount of virtual CPUs and Entitled Capacity. If Your processes are not multi threaded aware or they are not enough processes which can be scheduled, it should be helpful to disable SMT. ---
**Author:** ![MichaelFelt](https://community.unix.com/letter_avatar/michaelfelt/32/5_5575768a8748004e209b776fc1b2916d.png) [@MichaelFelt](https://community.unix.com/u/MichaelFelt)\ **Post date:** [November 23, 2012, 1:00pm UTC](https://community.unix.com/t/not-using-all-cpus/320231/10 "2012-11-23T13:00:54Z")
load average - what do you mean? The paging space # of 2.4% does not really mean much, because you have too much paging space to begin with. You can just turn them off the two extra ones with the swapoff command (and perhaps later rmps to remove them permanently). vpundit: I was not aware that the rest of UNIX had caught up with AIX - and using memory for file cacheing - good to hear. regarding the ratio of entitled capacity to virtual processors: your ratio of 1:10, while legal is not considered best practice. as mentioned "xRay" paying attention to SRAD information is important. use the mpstat command to see if you are using a single thread on each processor - if so, count the actual processors being activated, divide that number in two and use that as your number of VP - that should lower your physical processor cost. lparstat output is important to see "app" value - as an estimate of how busy the system as whole is busy. but in all honesty, there is so much that needs to be done, it is all patchwork until some proper settings are applied, and all testing restarted.