Slow ssh on Solaris 10 zone

skamal4u · March 22, 2018, 2:09pm

ssh is slow on solaris zone , and is getting stuck at the following place.

debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: Wrote 664 bytes for a total of 3325
Below is the ssh version being used: Sun_SSH_1.1.6, SSH protocols 1.5/2.0, OpenSSL 0x0090704f

Below is the sshd config file:

Protocol 2
UseDNS yes
PermitRootLogin yes
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
Subsystem sftp /xxx/openssh/sftp-server
AuthorizedKeysCommand /usr/local/bin/ldap_ssh_authorizedkeys
AuthorizedKeysCommandUser nobody

tried changing useDNS,GSSAPIAuthentication values to no and didn't help. Could someone please help with this problem ?

on global zone ssh is quick

hicksd8 · March 22, 2018, 2:22pm

Does that zone have its own dedicated network interface or is it sharing one with the global zone?

And, is it slow to connect or also slow when being used as well?

skamal4u · March 22, 2018, 2:51pm

zone doesn't have dedicated network interface , the zone is not slow in performance , the app running on it performing well , the load average is also normal . only issue is ssh . recently we moved to openssh & then issue started .

jim_mcnamara · March 22, 2018, 10:32pm

Why is usePAM turned on? Those are add-on object files that perform special tasks, and they are not nesessarily portable from one flavor of ssh to another.

Can you post the output of

ssh -vvv

from a connection attempt from another server to the problem zone?

skamal4u · March 22, 2018, 11:04pm

ssh -A xxxx -vvv
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to xxxx [x.x.x.x] port 22.
debug1: Connection established.
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/identity type -1
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/identity-cert type -1
debug3: Not a RSA1 key file /Users/praveen_sriperumbudhuri/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/id_rsa type 1
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/id_rsa-cert type -1
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/id_dsa type -1
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/id_dsa-cert type -1
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/id_ecdsa type -1
debug1: identity file /Users/praveen_sriperumbudhuri/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6
debug1: match: OpenSSH_6.6 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 4 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 864 bytes for a total of 885
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96
debug2: kex_parse_kexinit: hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 909
debug2: dh_gen_key: priv key bits set: 159/320
debug2: bits set: 1577/3191
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 416 bytes for a total of 1325
debug3: check_host_in_hostfile: host xxxxx filename /Users/praveen_sriperumbudhuri/.ssh/known_hosts
debug3: check_host_in_hostfile: host xxxxx filename /Users/praveen_sriperumbudhuri/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 4653
debug3: check_host_in_hostfile: host x.x.x.x filename /Users/praveen_sriperumbudhuri/.ssh/known_hosts
debug3: check_host_in_hostfile: host x.x.x.x filename /Users/praveen_sriperumbudhuri/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1690
debug1: Host �xxxx� is known and matches the RSA host key.
debug1: Found key in /Users/praveen_sriperumbudhuri/.ssh/known_hosts:4653
debug2: bits set: 1590/3191
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1341
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 52 bytes for a total of 1393
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: [*] yubikey-3504459 (0x55e00bb36f70)
debug2: key: /Users/praveen_sriperumbudhuri/.ssh/identity ((nil))
debug2: key: /Users/praveen_sriperumbudhuri/.ssh/id_rsa (0x55e00bb31110)
debug2: key: /Users/praveen_sriperumbudhuri/.ssh/id_dsa ((nil))
debug2: key: /Users/praveen_sriperumbudhuri/.ssh/id_ecdsa ((nil))
debug3: Wrote 84 bytes for a total of 1477
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: Trying to reverse map address x.x.x.x.
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1372003379' not found

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1372003379' not found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: [*] yubikey-3504459
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 388 bytes for a total of 1865
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: SHA1 fp 13:55:bb:e1:b8:2f:c6:2f:2c:1c:97:29:af:c7:1e:aa:7e:77:12:b1
debug3: sign_and_send_pubkey: RSA 13:55:bb:e1:b8:2f:c6:2f:2c:1c:97:29:af:c7:1e:aa:7e:77:12:b1
debug3: Wrote 660 bytes for a total of 2525
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug3: Wrote 136 bytes for a total of 2661
debug2: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 0: request auth-agent-req@openssh.com confirm 0
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env HOSTNAME
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env HISTSIZE
debug3: Ignored env SSH_CLIENT
debug3: Ignored env SSH_TTY
debug1: Sending env LC_ALL = C
debug2: channel 0: request env confirm 0
debug3: Ignored env USER
debug3: Ignored env LD_LIBRARY_PATH
debug3: Ignored env LS_COLORS
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env PS1
debug3: Ignored env HISTCONTROL
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug1: Sending env LANGUAGE = C
debug2: channel 0: request env confirm 0
debug3: Ignored env LOGNAME
debug3: Ignored env CVS_RSH
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env LESSOPEN
debug3: Ignored env G_BROKEN_FILENAMES
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: fd 4 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: Wrote 664 bytes for a total of 3325

Scrutinizer · March 23, 2018, 2:01am

Hi Jim, if usePAM is turned off then the PAM stack is bypassed altogether by sshd . I think this option is portable if as long as PAM simply exists on a particular Unix flavour and it should typically be switched on IMHO..

jim_mcnamara · March 23, 2018, 8:17am

hi Scrutinizer,
We had Solaris 10 issues with what turned out to be "foreign" PAM objects. This may not be the case here, you are correct.

Skamal4u -
I do not see anything except some RSA keys being rejected. Which may be okay.

Please define "slow", do you mean:

slow to connect initially - i.e., slow login
slow to transfer data
Both 1 & 2

Does the effect happen for all users?

With regard to the key - did you copy the key it is complaining about to or from a windows machine. Go to the .ssh directory for praveen_sriperumbudhuri. I think the file in question may have windows carriage control. Use the vi editor to check but do not change the key file - do you see ^M characters? If so, you can convert the file using dos2unix -as the the user preveen.

skamal4u · March 23, 2018, 1:42pm

hi to answer your question ,

it is only slow to connect , not slow to transfer . some time even to get password prompt it takes 10-15 minutes . the issue is with all the ldap account user , the local user logins is very quick , it cannot be ldap issue as the global zone works fine for those ldap users with no issues

hicksd8 · March 23, 2018, 1:48pm

Right, that answers the question that I asked in post#2.

My money is on the non-global zone having a DNS resolution issue (that the global zone doesn't have).

Check out your DNS settings for the non-global zone..

hicksd8 · March 23, 2018, 1:53pm

Is the non-global zone a sparse root zone or a whole root zone?

jim_mcnamara · March 23, 2018, 4:43pm

Finally - an answer to our questions. You may not understand, but most of us here have done UNIX work for a long time. What seems evident to you is not to us because we learned it was not always the same problem when looked at superficially.

So where on the network is the AD authenticator server? Is it in your solaris zone's subnet.

What I'm asking:
pretend the AD box is 10.10.20.123 and solaris box is on the very same rack so it is 10.10.20.155. This is what I'm asking.

It could be AD is 10.10.35.123 and solaris is 10.10.13.155 - different subnet.
You may need to look at /etc/resolv.conf

Hicksd8 is looking at the same question a different way. And it may not seem the same to you.