sftp + expect: disconnection & restart removes already transfered data.

Shell Programming and Scripting
1

I have an ftp statement that when it restarts, it will write over the top of the file at the other end, rather than append to the file part sitting at the destination.

This is a problem because the flaky connection fails so regularly that the 2GB file I try to transfer will never complete.

Do any of you guru's out there understand how to ensure sftp will to recognise the file part at the other end and append to it?

And while we're at it, how can I improve reliability? When I run in batch mode it dies after less then 2MB sent, however if I run from a logged in shell and run sftp off the commandline, it will survive almost 1GB before exits. Why might this be?

Here's the code so far (naturally I've got a ~/.ssh/config file that has the proxy settings of the other end, and FYI I use the "corkscrew" library for this)

#!/bin/ksh
set timeout 60

result=1
tries=0
# TODO while [[ $result -ne 0 ]] ; do
while [[ 1 -ne 0 ]] ; do
   expect << EOF
      spawn sftp me@host.name.com <<EOF
      expect "\*password:\*"
      send -- "${password}\r"
      expect "sftp> "
      send -- "lcd /path/to/local/file/\r"
      expect "sftp> "
      send -- "cd /remote/path/to/place/file/\r"
      expect "sftp> "
      send -- "put file_name.txt\r"
      expect "sftp> "
      send -- "bye\r"
EOF
   result=$?
   let tries+=1 ;
   echo "###result:${result}:tries bottom:${tries}"
done
2

You don't need to use expect, unless you just want to...

#!/bin/ksh

set -A A_FILE file1 file2 file3 file4 file5 file6 file7 file8 file9 file10

i=0

while [ -n "${A_FILE[$i]}" ]
do
   sftp -o IdentityFile=${KEYFILE} ${FTPUSER}@${FTPSERVER} <<-EOF
      put ${L_PATH}/${A_FILE[$i]} ${R_PATH}/${A_FILE[$i]}
      quit
   #
   # The EOF must be TABbed over.  It can not be spaces.
   # 
   EOF
   (( i = i + 1 ))
done

As for resuming a broken sftp session. I don't think sftp supports it,
but putty has pfstp and I think it does.

3

Unfortunately we can't use identity files because we don't have access to a login shell or a home directory. "expect" was the only way.

I did some further reading for high reliability file transfer and discovered the crafty Russians have designed THE swiss army knife of file transfer "lftp"

I'll install and try this and report back.:slight_smile: