I have a pretty demanding requirement for an SFTP script I have been trying to put together.
I have nearly 100 files (all with the names staring with T_PROD) generated in my local server daily. I need to transfer each of these files to a remote server via SFTP (that's a client demand, can't use SSH or others). Since these files are pretty big (approx 10 MB each), we need to have a check on succesful transfer of each of the files. In case a file is not transferred properly, we will be trying to re-send the file, at max 3 times.
Since it's SFTP, we're not allowed to execute shell commands in the remote server. I tried creating the md5sum hash values, but stuck thereafter. How do we identify the exact files those failed the sanity check, and try re-sending them?
A barebone structure of the script is as below
#!/bin/bash
. ./params # parameter file
cd output
sftp ${FTPUSER}@${FTPSERVER} << EOF
cd $FTPFOLDER
put T_PROD*
!md5sum * > chklist.md5
bye
EOF
I'm running this script in Linux kernel 2.6.39
Do let me know if any further information is required.
Your sftp session is not a normal shell session for running commands remotely so it is a little awkward. You command below
!md5sum * > chklist.md5
will run on the local server, so I'm not sure that is what you really want.
If you were using plain ftp then you would get a zero return code if the connection opened and you had to resort to switching on flags to get more verbose output that can be captured in a log file and read afterwards. sftp on the other hand is better with errors. Could you do something like the following instead?:-
#!/bin/bash
. ./params # parameter file
cd output
errcnt=0
for file in T_PROD*
do
sftp ${FTPUSER}@${FTPSERVER} << EOF
cd $FTPFOLDER
put $file
EOF
# Note not an indentation with spaces on previous line. Used tab for clarity.
if [ $? -ne 0 ]
then
printf "Something went wrong with file \"$file\"\n"
((errcnt=$errcnt+1))
else
printf "File \"$file\" send successfully.\n"
fi
done
if [ $errcnt -ne 0 ]
then
printf "ERROR in $errcnt transfers!\n"
done
I'd say sftp is a secure and safe protocol. It either succeeds or it doesn't. Run it with raised logging level (-v option) . Check the exit code after completion.
Hi achenle, as I've said in my first thread, using SFTP is a business requirement, and we can't help abiding by that.
Thanks for the help rbatte1. However, will just checking the return code of sftp ensure the correct transfer of files ? I'm dealing with file transfer things just recently [that's why I've used the md5sum, din't realise it worked only locally], and I believe the return code is 0 even if a file was transferred partially. Please correct me if I'm wrong.
I'll try out your suggestion though. Thanks a lot for the help again !!!
You could also list the file catching the output and comparing it later. Use an embedded dir $file in your script after you send the file. You can compare the number of bytes with the original file.
Unfortunately in this case sftp is pretty robust. I will see if I can test it out by sending a large file and then killing the sftpd process on the remote side to see what happens when I get a chance (unless anyone else would care to have a go first)