AIX 7.1 on Power 750 is very slow with no high memory or process consumption

UNIX AIX
1

Hi,

This thread has been posted before on linuxquestions.org, but no answer, maybe because this is unix question and not linux. I'm posting the same thread here, hope I can get an answer from someone in the meantime, I wish I could post of emergency thread but it needs bits which I don't have :

I'm posting this thread because first I'm a newbie on AIX (I've been a Linux Admin for almost 3 years and we have AIX for some project requirements) and second I could not find any useful info on that over the internet.
These servers are handed over to us by the client. We are experiencing slow performance on the console, on login (using Putty or any shell client on windows, the password pops up with a big latency) and on our Application Server (Oracle Weblogic 10.3.6). Here are the output of some of the usual commands :

-bash-4.2# svmon -G
               size       inuse        free         pin     virtual   mmode
memory     16842752     1459856    15382896     1291104     1392007     Ded
pg space    4325376        4707

               work        pers        clnt       other
pin          719376           0           0      571728
in use      1392007           0       67849

PageSize   PoolSize       inuse        pgsp         pin     virtual
s    4 KB         -      884208        4707      761888      816359
m   64 KB         -       35978           0       33076       35978

And here is the topas sample output:

Tue Mar  4 05:49:38 2014   Interval:2           Cswitch     240  Readch      190
                                                Syscall     130  Writech     400
CPU     User% Kern% Wait% Idle%   Physc  Entc%  Reads         1  Rawin         0
Total     0.0   0.0   0.0 100.0    0.01   0.07  Writes        1  Ttyout      188
                                                Forks         0  Igets         0
Network    BPS  I-Pkts  O-Pkts    B-In   B-Out  Execs         0  Namei         2
Total    3.68K   51.00    2.50   3.29K   396.0  Runqueue      0  Dirblk        0
                                                Waitqueue   0.0
Disk    Busy%      BPS     TPS  B-Read  B-Writ                   MEMORY
Total     0.0        0       0       0       0  PAGING           Real,MB   65792
                                                Faults        0  % Comp      8
FileSystem          BPS    TPS  B-Read  B-Writ  Steals        0  % Noncomp   0
Total             188.5   0.50   188.5       0  PgspIn        0  % Client    0
                                                PgspOut       0
Name           PID  CPU%  PgSp Owner            PageIn        0  PAGING SPACE
topas       6291672  0.0 3.51M root             PageOut       0  Size,MB   16896
reaffin      589854  0.0  640K root             Sios          0  % Used      0
swapper         524  0.0  448K root                              % Free    100
swapper         264  0.0  448K root             NFS (calls/sec)
getty       2491052  0.0  624K root             SerV2         0  WPAR Activ    0
java        3080656  0.0 78.8M root             CliV2         0  WPAR Total    0
sshd        6226168  0.0 1.23M root             SerV3         0  Press: "h"-help
gil         1769772  0.0  960K root             CliV3         0         "q"-quit

I even tried to check the bandwidth with a big file transfer and that looks fine.
Let me know if any other info is needed, actually we are stuck with this issue and we need to find a solution as soon as possible.

2

Hi and welcome to this forum,

to me it sounds like a DNS Timeout / Network misconfiguration problem - but i am not sure!

I would check the resolv.conf, /etc/hosts, etc. In this case a ssh -vvv can help to debug the problem.

I had a similar problem with the ssh daemon, in this case the ipv6 name resolutions caused big delays.

Regards

3

Hi,

this indeed sounds like a network configuration issue (dns). I saw this kind of behaviour before which was due to a dns misconfiguration.

Regards,

vincent

4

Well, actually we are using internal IPs instead of names, so that dns config seems a little bit strange to me.
Anyway, I've done ssh -vvv and here is the result :

OpenSSH_5.2p1, OpenSSL 0.9.8x 10 May 2012
debug1: Reading configuration data //.ssh/config
debug1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Failed dlopen: /usr/krb5/lib/libkrb5.a(libkrb5.a.so):   0509-022 Cannot load module /usr/krb5/lib/libkrb5.a(libkrb5.a.so).
        0509-026 System error: A file or directory in the path name does not exist.

debug1: Error loading Kerberos, disabling Kerberos auth.
debug2: ssh_connect: needpriv 0

debug1: Connecting to 192.168.110.45 [192.168.110.45] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /.ssh/identity type -1
debug3: Not a RSA1 key file /.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /.ssh/id_rsa type 1
debug1: identity file /.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.2
debug1: match: OpenSSH_5.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.2
debug2: fd 4 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: Entering the function :kex_choose_conf

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 129/256
debug2: bits set: 497/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /.ssh/known_hosts
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts
debug3: check_host_in_hostfile: filename /.ssh/known_hosts
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts
debug2: no key of type 0 for host 192.168.110.45
debug3: check_host_in_hostfile: filename /.ssh/known_hosts2
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts2
debug3: check_host_in_hostfile: filename /.ssh/known_hosts
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts
debug2: no key of type 2 for host 192.168.110.45
The authenticity of host '192.168.110.45 (192.168.110.45)' can't be established.
RSA key fingerprint is 93:c7:e8:e7:ce:31:42:82:1c:d8:c8:f5:2a:2d:79:57.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.110.45' (RSA) to the list of known hosts.
debug2: bits set: 546/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /.ssh/identity (0)
debug2: key: /.ssh/id_rsa (20084968)
debug2: key: /.ssh/id_dsa (0)
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: The Key: 0

debug1: Trying private key: /.ssh/identity
debug3: no such identity: /.ssh/identity
debug1: After function load_identity_file

debug1: The Key: 1

debug1: Offering public key: /.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: The Key: 0

debug1: Trying private key: /.ssh/id_dsa
debug3: no such identity: /.ssh/id_dsa
debug1: After function load_identity_file

debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: userauth_kbdint: disable: no info_req_seen
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred:
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@192.168.110.45's password:

Also, /etc/resolv.conf is as follows :

nameserver 192.168.110.200

(the nameserver above is for our Oracle RAC)

Can that be because of IPv6? actually I disabled IPv6 by

chdev -l lo0 -a netaddr6=''

command

5

At which point the SSH login hung? The Debug show no timestamps.

For a test with SSH vs. IPv6 you could enable the line ListenAddress 0.0.0.0 in the /etc/ssh/sshd_config and
restart the SSH-Daemon stopsrc -s sshd && startsrc -s sshd .

Regards

6

It may be trying to do reverse lookups on the IPs.

7

How can I try that? Actually

127/8              127.0.0.1         U         1      9559 lo0      -      -

I have that in my routing table and

-bash-4.2# ifconfig -a
en4: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 192.168.110.45 netmask 0xffff0000 broadcast 192.168.255.255
         tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
         tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

---------- Post updated at 10:11 PM ---------- Previous update was at 10:08 PM ----------

Here is when it hung:

debug1: Error loading Kerberos, disabling Kerberos auth.
debug2: ssh_connect: needpriv 0

and went on after a minute or so:

debug1: Connecting to 192.168.110.45 [192.168.110.45] port 22.