Remote side unexpectedly closed network connection + Solaris 10 ssh

Hi,
I have a Solaris 10 (sparc) server. Suddenly some of non-root user logins are stopped on this server. Local user root is able to login. Some of NIS accounts are working and some are not. All NIS accounts are fine, as I can login on other servers. NIS service also seems fine, as some users are still able to login.

Irrespective of I give correct password or incorrect password, it always fails with ssh/putty error - "Remote side unexpectedly closed network connection".
I tested it locally on server "ssh -vvvv asingh@localhost"
And that too fails with below message (last part is below)

debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: packet_send2: adding 32 (len 21 padlen 11 extra_pad 64)
Connection closed by 127.0.0.1
debug1: Calling cleanup 0x341a8(0x0)
nmsa-hpov-prod #

I ran truss also (attaching ssh.truss ssh.truss (142.8 KB)
) and I can't figure out what is issue

nmsa-hpov-prod # truss -afeld -vall -sall -o ~/ssh.truss ssh asingh@localhost
Password:
Connection closed by 127.0.0.1
nmsa-hpov-prod #

Pleae advise, how to figure it out

Thanks

Hello,

I think you might have better luck here doing the debugging at the server side, rather than the client side. The issue is not so much likely to be with the SSH client software, as it is with the SSH server. So you'd want to either try attaching a truss to your SSH server, or to run the SSH server directly at the command line in debug mode.

You can do this with OpenSSH by shutting down the SSH server service, and then typing a command like /usr/sbin/sshd -d -d -d as root. The SSH server will then run as a foreground process in your terminal window, and you can then try connecting the client and see what debug messages you are able to log on the server. The SSH server will then exit when the incoming connection closes. Just remember to start your SSH server service properly up again when you finish before you log out, or else you're going to be locked out

Hope this helps ! Let us know your thoughts, and how you get on, and we can then take things from there.

I tried this way and ssh gives me Segmentation Fault, if I login with non-root user.
Here is the output (sorry for long one)

-bash-3.2# /usr/lib/ssh/sshd -d -d -d
debug1: sshd version Sun_SSH_1.1
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: Bind to port 22 on ::.
Server listening on :: port 22.

debug1: Server will not fork when running in debugging mode.
Connection from 192.168.69.189 port 61585
debug1: Client protocol version 2.0; client software version PuTTY_Release_0.75
debug1: no match: PuTTY_Release_0.75
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.1
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: GSS-API Mechanism encoded as toWM5Slw5Ew8Mqkay+al2g==
debug1: SSH2_MSG_KEXINIT sent
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve448-sha512,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,rsa2048-sha256,rsa1024-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: kex_parse_kexinit: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-ed448,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-ctr,aes192-cbc,aes128-ctr,aes128-cbc,chacha20-poly1305@openssh.com,3des-ctr,3des-cbc,blowfish-ctr,blowfish-cbc,arcfour256,arcfour128
debug2: kex_parse_kexinit: aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-ctr,aes192-cbc,aes128-ctr,aes128-cbc,chacha20-poly1305@openssh.com,3des-ctr,3des-cbc,blowfish-ctr,blowfish-cbc,arcfour256,arcfour128
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-etm@openssh.com
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-etm@openssh.com
debug2: kex_parse_kexinit: none,zlib,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug2: mac_init: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: Peer sent proposed langtags, ctos:
debug1: Peer sent proposed langtags, stoc:
debug1: We proposed langtags, ctos: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug1: We proposed langtags, stoc: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: dh_gen_key: priv key bits set: 170/320
debug1: bits set: 517/1024
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: bits set: 510/1024
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug2: kex_derive_keys
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done


debug1: userauth-request for user ssrivast service ssh-connection method none
debug1: attempt 0 initial attempt 0 failures 0 initial failures 0
debug2: input_userauth_request: setting up authctxt for ssrivast
debug2: input_userauth_request: try method none
Failed none for ssrivast from 192.168.69.189 port 61585 ssh2
debug1: userauth-request for user ssrivast service ssh-connection method gssapi-with-mic
debug1: attempt 1 initial attempt 0 failures 1 initial failures 0
debug2: input_userauth_request: try method gssapi-with-mic
debug1: Client offered gssapi userauth with { 1 2 840 113554 1 2 2 } (supported)
debug1: userauth-request for user ssrivast service ssh-connection method keyboard-interactive
debug1: attempt 2 initial attempt 0 failures 1 initial failures 0
debug2: input_userauth_request: try method keyboard-interactive
debug1: keyboard-interactive devs
debug2: Starting PAM service sshd-kbdint for method keyboard-interactive
debug3: Trying to reverse map address 192.168.69.189.
debug2: Calling pam_authenticate()
debug2: PAM echo off prompt: Password:
debug2: Nesting dispatch_run loop

debug1: got 1 responses
debug2: Nested dispatch_run loop exited
debug1: PAM conv function returns PAM_SUCCESS
Segmentation Fault (core dumped)
-bash-3.2#

And here are my dump settings, if these matters

-bash-3.2# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 10
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 29995
virtual memory          (kbytes, -v) unlimited
-bash-3.2# coreadm
     global core file pattern:
     global core file content: default
       init core file pattern: core
       init core file content: default
            global core dumps: disabled
       per-process core dumps: enabled
      global setid core dumps: disabled
 per-process setid core dumps: disabled
     global core dump logging: disabled
-bash-3.2#

Not sure, where core is being dumped

-bash-3.2# dumpadm
      Dump content: kernel pages
       Dump device: /dev/dsk/c1t0d0s1 (swap)
Savecore directory: /var/crash/nmsa-hpov-prod
  Savecore enabled: yes
-bash-3.2# ls -ltr /var/crash/nmsa-hpov-prod
total 0
-bash-3.2#

On other side, it doesn't look like entirely ssh issue. Because there are some other NIS users, who are able to login well. Below is example from the user, when a NIS user was successfully able to login

-bash-3.2# /usr/lib/ssh/sshd -d -d -d
debug1: sshd version Sun_SSH_1.1
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: Bind to port 22 on ::.
Server listening on :: port 22.

debug1: Server will not fork when running in debugging mode.
Connection from 192.168.69.191 port 65477
debug1: Client protocol version 2.0; client software version PuTTY_Release_0.60
debug1: no match: PuTTY_Release_0.60
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.1
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: GSS-API Mechanism encoded as toWM5Slw5Ew8Mqkay+al2g==
debug1: SSH2_MSG_KEXINIT sent
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-ctr,aes192-cbc,aes128-ctr,aes128-cbc,blowfish-ctr,blowfish-cbc,3des-ctr,3des-cbc,arcfour256,arcfour128
debug2: kex_parse_kexinit: aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-ctr,aes192-cbc,aes128-ctr,aes128-cbc,blowfish-ctr,blowfish-cbc,3des-ctr,3des-cbc,arcfour256,arcfour128
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug2: mac_init: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: Peer sent proposed langtags, ctos:
debug1: Peer sent proposed langtags, stoc:
debug1: We proposed langtags, ctos: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug1: We proposed langtags, stoc: en-CA,es-MX,en-US,es,fr,fr-CA,i-default
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST_OLD received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: dh_gen_key: priv key bits set: 167/320
debug1: bits set: 538/1024
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: bits set: 503/1024
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug2: kex_derive_keys
debug3: kex_reset_dispatch -- should we dispatch_set(KEXINIT) here? 0 && !0
debug1: newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done

debug1: userauth-request for user vindulka service ssh-connection method none
debug1: attempt 0 initial attempt 0 failures 0 initial failures 0
debug2: input_userauth_request: setting up authctxt for vindulka
debug2: input_userauth_request: try method none
Failed none for vindulka from 192.168.69.191 port 65477 ssh2
debug1: userauth-request for user vindulka service ssh-connection method keyboard-interactive
debug1: attempt 1 initial attempt 0 failures 1 initial failures 0
debug2: input_userauth_request: try method keyboard-interactive
debug1: keyboard-interactive devs
debug2: Starting PAM service sshd-kbdint for method keyboard-interactive
debug3: Trying to reverse map address 192.168.69.191.
debug2: Calling pam_authenticate()
debug2: PAM echo off prompt: Password:
debug2: Nesting dispatch_run loop
debug1: got 1 responses
debug2: Nested dispatch_run loop exited
debug1: PAM conv function returns PAM_SUCCESS
debug2: kbd-int: pam_authenticate() succeeded
debug2: kbd-int: success (pam->state == 7)
Accepted keyboard-interactive for vindulka from 192.168.69.191 port 65477 ssh2
debug2: Monitor pid 7506, unprivileged child pid 7541
debug2: Monitor started
monitor debug3: Recording SSHv2 session login in wtmpx
monitor debug3: not writing utmpx entry
debug2: Waiting for monitor
debug2: Monitor signalled readiness
debug3: Setting handler to forward re-key packets to monitor
debug2: Unprivileged server process dropping privileges
debug1: permanently_set_uid: 30430/30430
monitor debug1: Entering monitor loop.
monitor debug1: fd 4 setting O_NONBLOCK
monitor debug1: fd 12 setting O_NONBLOCK
debug1: Entering interactive session for SSH2.
debug1: fd 3 setting O_NONBLOCK
debug1: fd 9 setting O_NONBLOCK
debug1: server_init_dispatch_20
debug3: server_init_dispatch_20 -- should we dispatch_set(KEXINIT) here? 1 && !0
debug3: server_init_dispatch_20 -- skipping dispatch_set(KEXINIT) in unpriv proc
debug1: server_input_channel_open: ctype session rchan 256 win 16384 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: init
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_pty_req: session 0 alloc /dev/pts/3
debug3: tty_parse_modes: SSH2 n_bytes 16
debug3: tty_parse_modes: 3 127
debug3: tty_parse_modes: ispeed 38400
debug3: tty_parse_modes: ospeed 38400
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
debug1: Setting controlling tty using TIOCSCTTY.
monitor debug3: writing utmpx entry
debug1: fd 4 setting TCP_NODELAY
debug1: fd 12 setting O_NONBLOCK
debug2: fd 11 is O_NONBLOCK
debug3: channel_set_wait_for_exit 0, 1 (type: 4)

^CMonitor killed; exiting <================I DID CTLR+C, WHEN USER WAS LOGGED IN
monitor debug1: Calling cleanup 0x459c0(0x0)
monitor debug1: Calling cleanup 0x27a94(0x79eac)
monitor debug1: Calling cleanup 0x27354(0x73000)
monitor debug1: Calling cleanup 0x1e92c(0xffbff76c)
monitor debug1: Calling cleanup 0x2627c(0x80238)
monitor debug1: Monitor received SIGCHLD.
monitor debug1: Calling cleanup 0x459c0(0x0)
-bash-3.2# 

Hello,

Thanks for that debug output. I'd agree this doesn't look strictly like an SSH server or client issue per se, but is more likely to be either related to NIS, PAM, or some aspect of the interconnectivity between the two.

The next question that comes to mind is: are the accounts which result on crashes in login always failing, or do they sometimes succeed ? And is it always the same accounts - i.e. is there no variability to this, and it is always the case that account X can log in, whereas account Y triggers a crash ? If that is the case, then the question becomes what is different between the setup of account X and account Y, since there may be some kind of difference that somehow triggers this issue.

It is always same users, who will be able to login and same users who will always fail on this server. If I check those users, I can see difference of password field. I have a guess/assumption. On my problem server, I may have very old ssh and new password algorithm may not be supported on it ?
In below example, first 2 users are able to login and later 2 users can't login. See the long password string in later one. Starting with $6 means SHA512

nmsa-hpov-prod # ypcat passwd | grep peteg
peteg:v.lguvaeH2qVE:55357:55357::/export/home/peteg:/bin/bash
nmsa-hpov-prod #
nmsa-hpov-prod # ypcat passwd | grep shellyp
shellyp:IiBODG.vFsTBE:30430:30430::/export/home/shellyp:/bin/bash
nmsa-hpov-prod #
nmsa-hpov-prod #
nmsa-hpov-prod # ypcat passwd | grep ssrivast
ssrivast:$6$x4WdY0x1EjlPqicK$evrN1aRpcjaejWObhr1VEqKzkn.vdlAI38WHkU5xJmzLr7F5b6cldWJi1u8Ki02DXUVUP5h4.4hrDGJzJv4150:55541:55541:Shashwat Srivastava:/export/home/ssrivast:/bin/bash
nmsa-hpov-prod #
nmsa-hpov-prod # ypcat passwd | grep asingh
asingh:$6$u2OVSos2UWCut747$voPIFlvq2HpG4V1SXwhfylqmUG9s2dER0TiHNBBNzf2DM0H/2jsFDZFb8yEmx9kK2R/T1ICYq6jnovBc08h7f/:30421:30421::/export/home/asingh:/bin/bash
nmsa-hpov-prod #
-bash-3.2# ssh -V
Sun_SSH_1.1, SSH protocols 1.5/2.0, OpenSSL 0x0090704f
-bash-3.2#

Hello,

That seems a reasonable hypothesis here, yes. It could very well be that the NIS users have a password using a complexity, length or algorithm that the PAM implementation on the Solaris 10 box does not support, and so it segfaults when trying to process them. Certainly there must be something about them that it doesn't like, or no longer does. If these users recently stopped working, it may also be worth inquiring as to whether or not this co-incided with a password change ? If so, that would be further evidence that this could be what you're dealing with here, yes.

Hi

I have seen this sort of thing before - the root user can login but others cannot.

I was working at Fujitsu.

A lot experts had been working on this problem but no luck.

The questions I asked were crucial - what has changed since the last time it was OK ??

Ventura lily someone mentioned that the Client has insisted on changing the permissions
Of all directories with the ‘t’ option !
A script was used to apply the change.

Unfortunately there is at least one directory where making that change was not locking
Down stuff but it should never be changed - /temp

So just in case login as root and check that directory and see what the documentation says about /temp permissions.

Good luck - I will monitor this page for a while and go through the other suggestions and diagnostics.

Kind Regards
Helpful

/tmp got changed to /temp by the spelling
Eventually also got changed

This error message sometimes happens when there is a issue with a cryptographic handshake.

Because some users have this issue and others do not (per @solaris_1977, but only vague details); one of the places to look is to make a list of the cryptographic applications (the apps that use cryto like SSL, etc) and make a list of the clients and the version number.

That is the approach I would consider, which is taking a step back and review the various clients and their respective version numbers.

Sometimes older SSL clients, for example, use weak crypto which can be rejected by more current processes, like SSHD, etc.

HTH

IMHO no crypto/key exchange problem in the ssh protocol, because it works for some users, and also would be indicated on the client side (ssh -v).

But I think you are right in suspecting a problem with long nis password hashes.
Long hashes should work with the local /etc/shadow but maybe not with the Solaris NIS client (library).
Apply latest Solaris patches; might add such support.
Activate the long pw hashes.

Hi @solaris_1977,

I made a diff from both logs and found two noticeable points:

• putty v0.60 was used with the working login, but v0.75 with the failed one. You can see differences in the cipher suites during the key exchange init resp. newer algorithms like curve*. What happens if you use the same older putty version on .69.189?
• In the failed case, gssapi-with-mic was tried. What happens if you disable it on the putty client (Connection > SSH > Auth > GSSAPI)?

Thanks for your suggestions. Here is what I did more.

I created a new test user on NIS master. By default it took SHA512 hashing, same as non-working users. I tried login to problem server and as expected, it was failed with "Remote side unexpectedly closed network connection".
I tried with same computer this time, to test it from same version of putty and changing gssapi-with-mic didn't help

-bash-3.2# ypcat passwd | grep ggould
ggould:$6$u2OVSos2UWCut747$voPIFlvq2HpG4V1SXwhfylqmUG9s2dER0TiHNBBNzf2DM0H/2jsFDZFb8yEmx9kK2R/T1ICYq6jnovBc08h7f/:55543:55543:Goeff Gould:/export/home/ggould:/bin/bash
-bash-3.2#

Next step - I copied password string from a working user (old hashing) to this new user. It worked.

-bash-3.2# ypcat passwd | grep moh
mmohamed:v.lguvaeH2qVE:55357:55357::/export/home/mmohamed:/bin/bash
-bash-3.2#
-bash-3.2# ypcat passwd | grep ggould
ggould:v.lguvaeH2qVE:55543:55543:Goeff Gould:/export/home/ggould:/bin/bash
-bash-3.2#

Here are SSH version and supported algo details on this problem server.

-bash-3.2# ssh -V
Sun_SSH_1.1, SSH protocols 1.5/2.0, OpenSSL 0x0090704f
-bash-3.2#
-bash-3.2# cat  /etc/security/policy.conf | grep -v "#"
AUTHS_GRANTED=solaris.device.cdrw
PROFS_GRANTED=Basic Solaris User

CRYPT_ALGORITHMS_ALLOW=1,2a,md5


CRYPT_DEFAULT=__unix__
-bash-3.2#

One interesting part is, irrespective of I give correct password or incorrect password, it will always fail with $6 password. So it looks like, it is not even reaching to level of matching authentication.
I am still try to find, if it is documented anywhere that my version of SSH/OpenSSL will not work with SHA512 users

That's too old.
Higher patch levels will change it to
CRYPT_ALGORITHMS_ALLOW=1,2a,md5,5,6
That'll work at least with /etc/shadow, and maybe with NIS (it could already fail because the transferred string exceeds the receive buffer).
I doubt that Oracle puts much effort into NIS (and NISplus), because their focus is LDAP.

Hi,

The problem here isn't with your SSH server, I don't think. Rather, it's the Solaris 10 implementation of PAM that is unable to cope with SHA512. Essentially, the SSH server is passing off to PAM for the authentication, and PAM is trying to decrypt the SHA512 salted password, and crashing when it does so. The SSH server is waiting for a "yes" or "no" from PAM, that answer never comes because PAM segfaults, and so the SSH server has no choice at that point but to drop the connection. So even if you were to grab the latest and greatest version of OpenSSH off the Web, compile it, and install it, you'd find the same error remained, I suspect.

Fundamentally, I think you have three options here. Firstly, you could look at upgrading the Solaris 10 server's software so that its version of PAM supports SHA512. I don't myself know if that's possible - maybe there are updates from Oracle to enable this, though you'd have to check that out.

Another thing you could try is switching the affected users to public key authentication, and not password authentication. If the SSH server never asks PAM to decrypt the password, the crash might never trigger. Again, that's not a sure thing, so you'd have to test that to see if it works (and this would have to be an acceptable option to your users, naturally). And you could find it difficult to find a key algorithm that was supported across all your estate, since it could be that the Solaris 10 SSH server only supports key types that are now deprecated on Linux and more modern releases of Solaris.

Lastly, you could simply ensure that all users that need to connect to the Solaris 10 server are using a password hashing algorithm for their NIS account that is supported by the Solaris 10 PAM implementation. You'd have to find an algorithm that worked on all your systems, if such a sweet spot can be found. And even then, it's entirely possible that as other clients (such as Linux systems) get updated, you might find the problem goes the other way - they could one day refuse to allow these accounts to connect because they perceive the stored password as insecure.

Hope this helps ! Let us know how you get on.

Hi,

Just discovered something that might actually be exactly what you're after. Oracle patch 140905-02 for Solaris 10 has the description "140905-02 – SunOS 5.10: sha256, sha512 patch". It appears to be concerned with, as the name implies, adding support for SHA256 and SHA512 password encryption.

So if you have access to Oracle support, that's definitely worth checking out, as patching your server up to and including this particular patch could resolve the issue.

Hi @solaris_1977

Honestly, as mentioned by me and others; you should do a inventory of your putty clients or whatever clients you are using first. It is important when solving a complex problem to define the problem first. You say "some clients work", "some clients do not work"; but you do not define what "some" means.

Please before we go further, just create a simple list, something like this:

OS           CLIENT                VERSION                 RESULTS

Windows 10     putty               0.3X                        FAIL
Windows 10     putty               0.76                        PASS
Ubuntu 20.x    ssh                 OpenSSH_8.1p1               PASS

I'm not trying to be a PITA, but when solving a problem like this; it is best to understand the scope of the problem before diving into the weeds. Normally, problems are easy to solve is we know what the problem is; but so far you have not actually defined your problem, leaving everyone here to just "guess" what is going on.

I have seen the problem before, and in each case (as mentioned earlier) the issue was because a client process was trying to negotiate a cryptographic handshake (agree on a cryptographic protocol) and the client was using some older crypto which is not supported by the server.

My experience is the quickest way to solve a problem is to understand the problem before trying to solve it. In this case, you have not defined the problem space; so everyone is guessing.

HTH.

A stock Solaris 10 has PAM logging off.
Turn it on in /etc/syslog.conf with

# SSH connections are logged to auth.info
auth.info<TAB>/var/adm/authlog

The <TAB> means one or more TAB characters - no space characters are allowed there.
Finally add a rotation for /var/adm/authlog to logadm.

I see this issue from all versions (which we have) of putty and I connection fails from problem server itself (localhost) and from other Solaris-9, Solaris-10, Solaris-11, Linux, ubuntu server. So it looks like it is not about putty version of the source, from where I am connecting.
So, for now I think I am going with ssh patch update (or install recommended patch cluster, which would upgrade other patches too) as drysdalk and MadeInGermany suggested.
For now, I will consider this thread as closed after getting suggestions. I will take some time to get downtime of this server and apply this solution. Thanks all for amazing help.