AIX 6.1 Memory Performance issues

UNIX AIX
1

Good Day Everyone,
Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for each server). From our observation, after ETL jobs run for quite sometime (about a week), the performance of jobs start degrading. And we found out that the paging start occurring. After that we decided to restart all 3 servers after the whole day batch job finished. After restart, for the next 5-7 days the performance is within our expectation, but after that the paging start happen again.
Anyone has shine me bright idea how to resolve this or anyone encounter this before ? your input is very much appreciated.

Thank you Everyone!

2

Using which tool did you see the paging (just to make sure)?
Do you have a long term monitoring set up like with nmon so that you can see when the Paging Out occurs?
You could write a short script and put it to the crontab or background on those 3 machines with the paging, so that you can note down every 10 minutes the size of the processes. Maybe you have a memory leak that uses up RAM until things have to be swapped out, most probably reducing the cached files first.

3

Hi zaxxon,
Yes I have nmon capture running all the time (daily, weekly and monthly). From the nmon I can find the Paging happen.

Thanks for the reply.

Best Regards,
ckwan

4

You could put something like this in a cronjob:

$ printf "DATE: %s\n" "$(date)"; svmon -P| awk '/^[[:space:]]*Pid/ {getline; print $1, $2, $5} END{print "-----------------"}'
DATE: Tue Jul  2 11:15:27 CEST 2013
9568460 sshd 0
10747986 ksh 0
6684732 awk 0
10485876 svmon 0
-----------------

1st is the PID, 2nd the process name and 3rd pages in paging space.
Redirect >> it to some logfile and have it run every 10 minutes, just for monitoring.
As root you will see many more processes - so you can just filter those of interesst for you.