running script in cron - with ssh commands - not working

I ran an ssh command to run a script on a remote server

ssh -l <user> <servername> /path/to/script/scriptname

This works fine - and the script is executed correctly.

However - I put this command into a script, that I want to run from cron every hour, to execute the file on the remote server every hour, but the process appears to hang... I can see the process but it is not doing anything. It only takes 3-4 seconds to run from command line but doesnt do anything when cron submits it:
root 16528 16527 0 14:46:00 ? 0:00 /usr/bin/ksh /<pathtoscript>/<script>
root 16527 233 0 14:46:00 ? 0:00 sh -c /<pathtoscript>/<script> > /dev/null
root 16532 16528 0 14:46:00 ? 0:00 ssh -l <user> <server> //<pathtoscript>/<script>

This just hangs.... Any ideas?

Have you tried to debug this? Check out your cron log. For every job that is executed, an entry is made, along with the pid of the process that is spawned for this job. search for the entry of the cronjob and then the corresponding process. i.e

root 13151 c Mon May 2 23:55:00 IST 2005
CMD: /opt/cron/load.sh
root 13151 c Mon May 2 23:55:00 IST 2005 rc=2

The cronjob specified above has failed.

Can you check some entries like that in your cron logs?

If the cron job goes through, then the last rc=[errno] will not be there.

With ssh, important things to check:

1. is the cronjob maintained for the correct user
2. is the key maintainence done on remote end (cron will not allow user interaction and the login will have to be key based authentication)

Thanks for looking in to this for me.. Still havent found the issue yet..

I checked the cron log:
> CMD: /pathtoscript/script.sh 2>/dev/null
> root 515 c Sat May 7 13:33:00 2005
< root 513 c Sat May 7 13:33:00 2005

Only when I kill the first ssh script that is trying to run - is the cron log updated again. I waited 5 mins before killing the script that wasnt doing anything, then cron log got this entry:

< root 515 c Sat May 7 13:33:53 2005 rc=143

My crontab is for my root user, which can run the script from the command line but not in cron. Key Maintainance, I think this is done from my end. I generated the keys and sent the public key to the others servers authorized_keys file. It works from command line and doesnt require password to be entered. This is why I dont know why cron cannot run it.

Any other ideas?

Try running this in a script that looks like this -

#!/bin/ksh
ssh -l <user> <servername> /path/to/script/scriptname
exit

Instead of the ssh command, use the script in the cron. It may be that cron is using /bin/sh to run commands, while you are running this from ksh. It should not really matter, but sometimes it does.

This is what script looks like - I think I am already going what you are asking...

#!/usr/bin/ksh

current_date=`date '+%Y%m%d'`
date_time=`date '+%Y%m%d %H:%M'`

### Execute remote Scripts ###

ssh -l <user> <servername> "/path/to/script/scriptname"
ssh -l <user> <servername2> "/path/to/script/scriptname"
ssh -l <user> <servername3> "/path/to/script/scriptname"
ssh -l <user> <servername4> "/path/to/script/scriptname"
ssh -l <user> <servername5> "/path/to/script/scriptname"
ssh -l <user> <servername6> "/path/to/script/scriptname"

You said you are using root's cron. Does this mean you have public-key authentication set up as root?

When you run the script from the command line are you running it as root also or another user?

Yes - I have public-key authentication set up as root.
However - the user I am running the script as on the other server is not called root. ie. Root on my server, user1 on the other server

I run the ssh script from command line on my server as root user and it works fine. (ssh -l <user1> <servername> "/path/to/script/scriptname"

try changing your cron to output stdout and stderr to a log:

for example

30 * * * * ssh -l user1 servername /path/to/script/scriptname > /tmp/sshcron 2>&1

What do you see in the output file?

Tried that - but nothing is logged to the file. The file is created but nothing errors to the file. The process appears to hang.. I left it for 10 mins then killed it. Then only thing that is written to the log file is:
Killed by signal 15

I think that the problem is it's waiting for you to type a password. I'm having this same issue, I can't figure out how to pass the password to it in a script.

This shouldnt be the issue as key authentication is in place and I can run the script from the command line without being asked for passwords..

Why should this make a difference from cron?

Can you change the ssh command line to include -v.

If that doesnt help try to truss the process.

I included the -v option - Again I had to kill the processes after 10 mins. I have pasted theoutput below. Can you see what is wrong?

SSH Version Sun_SSH_1.0, protocol versions 1.5/2.0.
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: ssh_connect: getuid 0 geteuid 0 anon 0
debug1: Connecting to server1 [10.192.1.160] port 22.
debug1: Allocated local port 1023.
debug1: Connection established.
debug1: identity file //.ssh/identity type 3
debug1: Bad RSA1 key file //.ssh/id_rsa.
debug1: identity file //.ssh/id_rsa type 3
debug1: Bad RSA1 key file //.ssh/id_dsa.
debug1: identity file //.ssh/id_dsa type 3
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.8.1p1
debug1: match: OpenSSH_3.8.1p1 pat ^OpenSSH
Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.0
debug1: sent kexinit: diffie-hellman-group1-sha1
debug1: sent kexinit: ssh-rsa,ssh-dss
debug1: sent kexinit: aes128-cbc,blowfish-cbc,3des-cbc,rijndael128-cbc
debug1: sent kexinit: aes128-cbc,blowfish-cbc,3des-cbc,rijndael128-cbc
debug1: sent kexinit: hmac-sha1,hmac-md5
debug1: sent kexinit: hmac-sha1,hmac-md5
debug1: sent kexinit: none
debug1: sent kexinit: none
debug1: sent kexinit:
debug1: sent kexinit:
debug1: send KEXINIT
debug1: done
debug1: wait KEXINIT
debug1: got kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug1: got kexinit: ssh-rsa,ssh-dss
debug1: got kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-c
tr,aes192-ctr,aes256-ctr
debug1: got kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-c
tr,aes192-ctr,aes256-ctr
debug1: got kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug1: got kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug1: got kexinit: none,zlib
debug1: got kexinit: none,zlib
debug1: got kexinit:
debug1: got kexinit:
debug1: first kex follow: 0
debug1: reserved: 0
debug1: done
debug1: kex: server->client unable to decide common locale
debug1: kex: server->client aes128-cbc hmac-sha1 none
debug1: kex: client->server unable to decide common locale
debug1: kex: client->server aes128-cbc hmac-sha1 none
debug1: Sending SSH2_MSG_KEXDH_INIT.
debug1: bits set: 496/1024
debug1: Wait SSH2_MSG_KEXDH_REPLY.
debug1: Got SSH2_MSG_KEXDH_REPLY.
debug1: Host 'server1' is known and matches the RSA host key.
debug1: Found key in //.ssh/known_hosts:2
debug1: bits set: 525/1024
debug1: ssh_rsa_verify: signature correct
debug1: Wait SSH2_MSG_NEWKEYS.
debug1: GOT SSH2_MSG_NEWKEYS.
debug1: send SSH2_MSG_NEWKEYS.
debug1: done: send SSH2_MSG_NEWKEYS.
debug1: done: KEX2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug1: service_accept: ssh-userauth
debug1: got SSH2_MSG_SERVICE_ACCEPT
debug1: authentications that can continue: publickey,password,keyboard-interactive
debug1: next auth method to try is publickey
debug1: key does not exist: //.ssh/identity
debug1: try pubkey: //.ssh/id_rsa
debug1: read SSH2 private key done: name rsa w/o comment success 1
debug1: ssh-userauth2 successfull: method publickey
debug1: fd 6 setting O_NONBLOCK
debug1: fd 7 setting O_NONBLOCK
debug1: fd 8 IS O_NONBLOCK
debug1: channel 0: new [client-session]
debug1: send channel open 0
debug1: Entering interactive session.
debug1: client_init id 0 arg 0
debug1: Sending command: /pathtoscript/script.sh
debug1: channel 0: open confirm rwindow 0 rmax 32768
debug1: channel 0: read<=0 rfd 6 len 0
debug1: channel 0: read failed
debug1: channel 0: input open->drain
debug1: channel 0: close_read
debug1: channel 0: input: no drain shortcut
debug1: channel 0: ibuf empty
debug1: channel 0: input drain->closed
debug1: channel 0: send eof
debug1: channel 0: rcvd eof
debug1: channel 0: output open->drain
debug1: channel 0: obuf empty
debug1: channel 0: output drain->closed
debug1: channel 0: close_write
debug1: channel 0: send close
debug1: channel: 0 rcvd request for exit-status
debug1: cb_fn 2677c cb_event 91
debug1: channel 0: rcvd close
debug1: channel 0: full closed2
debug1: channel_free: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i8/0 o128/0 fd -1/-1)

debug1: channel_free: channel 0: dettaching channel user

Anyone able to help out here at all?
Any one see anything in the ssh debug that suggests why it wont run correctly?

From your logs, it is clear that the login is taking place:

The problem is probably with the command/script that you are running on the remote system. What are you running on the remote system? Can you change the remote command to something like 'touch /tmp/testfile' - basically anything to confirm that login is taking place on remote system?

Thanks for the reply.

Just tried that. Basically i changed the script on the 2 remote servers to just touch a file in a directory on the remote server.

I then let cron run the script on my server to execute the script with the change above.

The file was created on the remote server 1 but hasnt got to the 2nd server yet- as far as my server is concerned, its still executing the remote script on server1.. when I do a ps -ef | grep for the script I get the following:

bash-2.05# ps -ef | grep daily
root 27707 27701 0 10:43:00 ? 0:00 ssh -v user@server1 /pathtoremotedir/script.sh root 27701 27698 0 10:43:00 ? 0:00 /usr/bin/ksh /localserver/script.sh root 27698 231 0 10:43:00 ? 0:00 sh -c /localserver/script.sh > /tmp/sshcron 2>&1

The last few lines in my debug after this are:
debug1: cb_fn 2677c cb_event 91
debug1: channel 0: rcvd close
debug1: channel 0: full closed2
debug1: channel_free: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i8/0 o128/0 fd -1/-1)

debug1: channel_free: channel 0: dettaching channel user

Idea's?

we have all 3 variants of your cron jobs in our cron tables and they all work fine ...

if the command runs normally on the command line but doesn't run properly through cron --- i don't think the issue is with the keys itself (of course, i could be wrong too) ... you may have a process that is looking for input or at least checking stdin/stdout/stderr or having the incorrect path or checking some environment variable that is not being set in your script prior to it running in cron ... maybe you can run a script in cron that just records the output from "env" and then you can compare that to the output of "env" in your interactive login ...

btw, you might want to check both the local and the remote scripts ...

I tried this now. I compared cron env and interactive env. There were differences so I set these in my script to be set before I run remote ssh... No difference in processing script - I still hit the same issue.

I also checked the env settings on the remote server when I run the script from cron or int - only difference is the port that it used when SSH connects to execute the script. I am at a loss...

in that case, try echoing all the variables and their values on the remote script to a file you can read when it is run from cron ... see if the same variables and values are showing up in the interactive run ... from what i gather from all the posts, there's a line in your remote script that's making it hang --- my experience says this is most likely something that needs to read a file ...

btw, what does the remote script do?

They all appear to be the same.

I was getting the remote script to check processes running etc.
But I have removed all of this and changed the remote script to just touch a file... thats all. And it still hangs.. although the file is created the ssh cron job on my side appears to be hanging...