Newbie - AIX LPAR performance problem

Running into performance issues with WAS application servers on two of LPAR's or like configuration under high load web pages crawl.

Please forgive me I'm new to AIX and most my expertise is in the Linux space.
Thanks for your help!!

Here's the run down:
The problem appears to be CPU related just don't know where... maybe entitlement??? There is 0 paging going on according to vmstat.
The run queue is .5625 based from 9 running threads and 16 total threads according to vmstat.
There is 0 waits and idle at its lowest is 26%. user processes 72% and 2% system.

Here is output from sar:

System configuration: lcpu=16 ent=4.00 mode=Capped 
cpu     %usr    %sys    %wio    %idle   physc   %entc
0       88      5       0       7       0.33    8.3
1       83      2       0       15      0.26    6.5
2       63      1       0       35      0.14    3.6
3       63      1       0       36      0.14    3.6
4       92      3       0       5       0.37    9.2
5       83      2       0       15      0.28    6.9
6       59      2       0       39      0.14    3.4
7       58      1       0       41      0.14    3.4
8       91      3       0       6       0.35    8.7
9       83      2       0       15      0.27    6.7
10      64      2       0       35      0.14    3.6
11      59      1       0       40      0.13    3.3
12      91      3       0       6       0.32    8.0
13      87      2       0       11      0.28    6.9
14      58      1       0       41      0.13    3.2
15      57      2       0       41      0.12    3.1
U       -       -       0       12      0.47    11.8

Here is the output from lpartstat -i

Node Name : node1
Partition Name : node1
Partition Number : 4
Type : Shared-SMT-4
Mode : Capped
Entitled Capacity : 4.00
Partition Group-ID : 32772
Shared Pool ID : 0
Online Virtual CPUs : 4
Maximum Virtual CPUs : 8
Minimum Virtual CPUs : 2
Online Memory : 16384 MB
Maximum Memory : 24576 MB
Minimum Memory : 8192 MB
Variable Capacity Weight : 0
Minimum Capacity : 2.00
Maximum Capacity : 8.00
Capacity Increment : 0.01
Maximum Physical CPUs in system : 64
Active Physical CPUs in system : 18
Active CPUs in Pool : 18
Shared Physical CPUs in system : 18
Maximum Capacity of Pool : 1800
Entitled Capacity of Pool : 1400
Unallocated Capacity : 0.00
Physical CPU Percentage : 100.00%
Unallocated Weight : 0
Memory Mode : Dedicated
Total I/O Memory Entitlement : -
Variable Memory Capacity Weight : -
Memory Pool ID : -
Physical Memory in the Pool : -
Hypervisor Page Size : -
Unallocated Variable Memory Capacity Weight: -
Unallocated I/O Memory entitlement : -
Memory Group ID of LPAR : -
Desired Virtual CPUs : 4
Desired Memory : 16384 MB
Desired Variable Capacity Weight : 0
Desired Capacity : 4.00
Target Memory Expansion Factor : -
Target Memory Expansion Size : -
Power Saving Mode : Disabled

Here is the output from smtctl

This system is SMT capable.
This system supports up to 4 SMT threads per processor.
SMT is currently enabled.
SMT boot mode is not set.
SMT threads are bound to the same virtual processor.
proc0 has 4 SMT threads.
Bind processor 0 is bound with proc0
Bind processor 1 is bound with proc0
Bind processor 2 is bound with proc0
Bind processor 3 is bound with proc0
proc4 has 4 SMT threads.
Bind processor 4 is bound with proc4
Bind processor 5 is bound with proc4
Bind processor 6 is bound with proc4
Bind processor 7 is bound with proc4
proc8 has 4 SMT threads.
Bind processor 8 is bound with proc8
Bind processor 9 is bound with proc8
Bind processor 10 is bound with proc8
Bind processor 11 is bound with proc8
proc12 has 4 SMT threads.
Bind processor 12 is bound with proc12
Bind processor 13 is bound with proc12
Bind processor 14 is bound with proc12
Bind processor 15 is bound with proc12

You need to provide more information. Post corresponding output from the vmstat command for several runs of the vmstat command. The only thing this really shows is that it looks like user code is consuming most of the cycles. Yet you're getting idle percentage.

Post actual process/thread information as well. What are these threads.

When "web pages are slow" are these browsers actually connected to the WAS servers or are the WAS servers strictly back end servers....use neststat-an. What are the ports states how many transactions shown if any in the send and receive queues.

What does iostat reveal?

What piece of code are the top threads running (see dbx or kdb)?

Next time, post output like this...

#System configuration:  lcpu=16 ent=4.00        mode=Capped
cpu     %usr    %sys    %wio    %idle   physc   %entc
0       88      5       0       7       0.33    8.3
1       83      2       0       15      0.26    6.5
2       63      1       0       35      0.14    3.6
3       63      1       0       36      0.14    3.6
4       92      3       0       5       0.37    9.2
5       83      2       0       15      0.28    6.9
6       59      2       0       39      0.14    3.4
7       58      1       0       41      0.14    3.4
8       91      3       0       6       0.35    8.7
9       83      2       0       15      0.27    6.7
10      64      2       0       35      0.14    3.6
11      59      1       0       40      0.13    3.3
12      91      3       0       6       0.32    8.0
13      87      2       0       11      0.28    6.9
14      58      1       0       41      0.13    3.2
15      57      2       0       41      0.12    3.1
U       -       -       0       12      0.47    11.8
~

Sometimes WAS give memory leaks especially with AIX. a bounce will be okay.

It takes us hours and hours to perform analysis. but the WAS just needs a little hug. =)

A bit more about why/what makes this a performance problem.

p.s. it is not that you do not have enough entitlement (in the output you show us) because that measurement shows a U column, which is unused entitlement.

So, was the data you sent from when the application is "failing", or just when you typed it in.

Assuming you have IBM support for both WAS and AIX I would open an initial call for support. It does not matter that you are new to AIX. They will ask you to make a snap when the system is running "not to your satisfaction" and come back with suggestions.

The advantage of using a tool like snap , or perfpmr , is that the data collection is standardized.