FTP/SFTP/SCP Files

Shell Programming and Scripting
1

Hi,

I have a process which FTP's the files from one server to another server.
Sometimes only half or a part of the file is delivered to remote location, but on the end log says FTP is successful. But ideally file in full is not delivered to remote location.
How can i catch these kind of errors in FTP as well as in SFTP?
How can i make sure file is delivered in full to remote location.
Any Checksum or calculating file size of remote location would help?

2

For file transfers "ftp" or "sftp" is usually a last resort when transferring files to incompatible or isolated platforms. If you have the opportunity to use "rcp" or "scp" backed with processing in Remote Shell this is preferable.

Assuming you only have "ftp" or "scp" the common techniques are:
1) Transfer each file under a temporary name and then rename the files to their correct names after the last transfer has finished.
2) Include a verifiable control record in the data file design.
3) Transfer a batch header file first which itemises filenames and checksums for the remote computer to check. (This assumes some compatibility between the computers).
4) Compress the file before transfer into a verfiable archive.

Obviously your "ftp" script needs proper error handling but I'd first look at why the transfers are failing and then how to deal with it. In my experience the most common "ftp" problem is the "mysterious hang" or "crawling speed" when transferring across a modern high-speed WAN.

Ps. Just scanned some of your earlier posts. Can you post the actual script because I see no error handling?

3

Thanks for the response. Please see my FTP code snippet below

for trial in {1..5} # Retry FTP 5 times
   do
     echo -e "\nAttempt to FTP the file ==> $trial" >> $upload_log;
     /usr/bin/ftp -niv $ftp_host > ${work_dir}/${abc}_ftp_temp.txt 2>&1 <<EndFtp
      quote user $ftp_user
      quote pass $ftp_pass
      cd $ftp_dir
      bin
      put $upload_file_name
EndFtp
      cat ${work_dir}/${abc}_ftp_temp.txt >> $upload_log;
      egrep -ri -h "^226" ${work_dir}/${abc}_ftp_temp.txt > /dev/null; 
# Transfer Check
      found=`echo $?`;
      if [ $found -eq 0 ]; then 
 # If FTP is successfull, break the loop and try FTP for next file
         break;
         elif [ $trial -lt 5 ]; then
           sleep 60;  # Halt the process for a minute
      fi
  done

FTP log is written to a temp file. After FTP is complete, code will search for string starting with 226 ( FTP Code for transfer success ) and if not available, then do a FTP retry. If 226 is available, loop is exited.

Here the problem is 226 code is possible for half transferred files too..So i am trying to find a way to find out part of the file transferes and re-transfer them.