Problem with ssh on target server.

Shell Programming and Scripting
1

ssh works from source server srcuser@10.8.44.13 to all other target servers except one which is target server trguser@10.8.44.43

On target the <trguser-home>/.ssh folder is set to permission 700 and authorized_keys file is set to permissions 600

cksum for id_rsa.pub on source 10.8.44.13 and authorized_keys on target is the same and the same has been verified to be good using cat -ev <filename>

The debug for failing ssh is as below.

ssh -vvvv trguser@10.8.44.43
OpenSSH_6.0p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Failed dlopen: /usr/krb5/lib/libkrb5.a(libkrb5.a.so): Could not load module /usr/krb5/lib/libkrb5.a(libkrb5.a.so).
System error: No such file or directory

debug1: Error loading Kerberos, disabling Kerberos auth.
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.8.44.43 [10.8.44.43] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/wd/srcuser/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /wd/srcuser/.ssh/id_rsa type 1
debug1: identity file /wd/srcuser/.ssh/id_rsa-cert type -1
debug1: identity file /wd/srcuser/.ssh/id_dsa type -1
debug1: identity file /wd/srcuser/.ssh/id_dsa-cert type -1
debug1: identity file /wd/srcuser/.ssh/id_ecdsa type -1
debug1: identity file /wd/srcuser/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "10.8.44.43" from file "/wd/srcuser/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /wd/srcuser/.ssh/known_hosts:183
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group14-sha1,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,gss-gex-sha1-,gss-group14-sha1-
debug2: kex_parse_kexinit: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA eb:d3:81:e8:25:7c:31:6a:0d:13:02:07:68:5d:7f:70
debug3: load_hostkeys: loading entries for host "10.8.44.43" from file "/wd/srcuser/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /wd/srcuser/.ssh/known_hosts:183
debug3: load_hostkeys: loaded 1 keys
debug1: Host '10.8.44.43' is known and matches the ECDSA host key.
debug1: Found key in /wd/srcuser/.ssh/known_hosts:183
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /wd/srcuser/.ssh/id_rsa (200631d8)
debug2: key: /wd/srcuser/.ssh/id_dsa (0)
debug2: key: /wd/srcuser/.ssh/id_ecdsa (0)
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /wd/srcuser/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /wd/srcuser/.ssh/id_dsa
debug3: no such identity: /wd/srcuser/.ssh/id_dsa
debug1: Trying private key: /wd/srcuser/.ssh/id_ecdsa
debug3: no such identity: /wd/srcuser/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
trguser@10.8.44.43's password:

I tried to create authorized_keys outside the home directory on the target server and modify the sshd configuration to point to use this authorized keys and restarted the sshd service but that too did not help.

I can share a successful ssh from the same source to a different target if that helps debug the issue.

Can you please suggest what could be the issue.

2

How about permissions on $HOME folder of user ( <trguser-home>) ?
Is it writable by others ?

Regards
Peasant.

3

The permissions for the HOME folder for working targets and non-working one is the same i.e. 755

4

Any clues / suggestions please?

ls -ltrd /app/trguser
drwxr-xr-x. 25 trguser trguser 4096 Aug 12 17:54 /app/trguser

uname -a
Linux targethost 3.10.0-957.21.3.el7.x86_64 #1 SMP Fri Jun 14 02:54:29 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
5

Try to ssh in from other servers on the network and see if you can isolate the problem to the client-side, the network, or the server configuration.

That is my suggestion.... trouble shoot your problem in a step-by-step manner.

6

Check the log of the ssh server and increase log level of the ssh server before that to DEBUG/DEBUG2.

I'm sorry I didn't read the ssh debug output. It's difficult to read on my very small mobile phone.

Did you compare the debug output of the failed ssh-client with those which are working?

7

I could not figure out the problem.

Here is a working ssh Debug:

8

Stomp suggested you turn on the server side debug.
Please do so, and check the logs from server side - the side you are connecting to.

Regards
Peasant.

9

Hi Mohtashims,

I'm now sitting on my notebook and I checked the two different Debug-Outputs of the 2 ssh connections. I clearly see differences.

I assume, you have - as I suggested - to debug the server side. But from the client side, there are visible differences, which shed a little light on the situation.

General Hints

I'm watching you and your questions quite a while. Your technical knowledge is quite good. You have scripting skills. You know how to use the commands and build programs upon that. That's very good.

What I see too, is that you are quickly asking questions if you stumble upon errors like here. So I think a further good skill, which will bring you forward in your overall competence would be problem investigation and problem solving capability.

What do you need for that?

  • Patience and willingness to deeply and carefully investigate a problem. Take your time, read debug output very carefully.
  • Try to understand what's happening! Step by step.
  • Think about it! What can be wrong? Where can be an error?
  • Think about how to get more information in the process! Where to get more information in the process?
  • Isolate more and more exactly the spot of the error
  • Debug output your variables
  • Make small test programs

    If you do not understand why the whole program reacts like it does, take small bits out of it and make a small testing program and take the problematic data into that.
  • Be prepared for own blind spots.

    Program errors you do not recognize even if they are directly in front of you, even if you'd seen them a dozen of times.(I had two spelling errors in your name in this post even if I read a hundred times in this forum so far.) Maybe Programming language behaviour that is not like you always believed it to be.
  • From time to time a break helps a lot

Apply that on the current problem and you will get closer to the cause.

So my question to you is here: What are the differences between the two client connection session debug logs?