SFTP Shell Script Get & Delete && Upload & Delete

Shell Programming and Scripting
1

Hi All,

Do you have any sample script,

  • auto get file from SFTP remote server and delete file in remove server after downloaded.
  • only download specify filename
  • auto upload file from local to SFTP remote server and delete local folder file after uploaded
  • only upload specify filename

This is my first shell script get files from sftp, it look no prefect:

#!/bin/bash

while [ 1 ]
do
        sftp -o IdentityFile=/root/.ssh/xxx xxx@10.xx.xx.xx << EOF
        cd outbound_test
        lcd /home/AAA/
        get AAA*
        rm AAA*
        lcd /home/BBB/
        get BBB*
        rm BBB*
        lcd /home/CCC/
        get CCC*
        rm CCC*
        !
EOF
        #sleep 300sec
        sleep 300
done
2

Most important question first: Does your script do what it is intended for? If not, post what goes wrong. If yes, here's some comments on it:

  • do you really need to run it continuously, i.e. will any of the files AAA*/BBB*/CCC* be recreated periodically?
  • while [ 1 ] works (creating an infinite loop) as would while [ 0 ] . Is that what you intended to do?
  • you want a download as well as an upload. In the same script? Switchable? File names changing?
  • you want "specified filenames" to be down-/uploaded. This can be easily done as lines in "here documents" are expanded by the shell. Any idea on how to supply the filenames?
  • adding comments is good behaviour in programming, but try to be more illustrative, don't state the obvious.
3

Most important question first: Does your script do what it is intended for? If not, post what goes wrong. If yes, here's some comments on it:

  • do you really need to run it continuously, i.e. will any of the files AAA*/BBB*/CCC* be recreated periodically?
    YES, customer will upload filename with AAA_%% or BBB_%%...
  • while [ 1 ] works (creating an infinite loop) as would while [ 0 ] . Is that what you intended to do?
    YES, I want run this as Services
  • you want a download as well as an upload. In the same script? Switchable? File names changing?
    Can split in 2 script
  • you want "specified filenames" to be down-/uploaded. This can be easily done as lines in "here documents" are expanded by the shell. Any idea on how to supply the filenames?
    It is random filename, only know the prefix name
  • adding comments is good behaviour in programming, but try to be more illustrative, don't state the obvious.

---------- Post updated at 06:22 AM ---------- Previous update was at 06:16 AM ----------

After 8hours research, I've this auto download script:


#!/bin/bash
Server=".ssh/xxxxx@10.x.x.x"
Logfile="ftp_log.log"
Processed_Logfile="ftp_log.processed"
Script_Logfile="ftp_log.script"
Directory="test"
Sleep=300

# -----------------------------------------------------
# Function get and remove file from server
# -----------------------------------------------------
getdelete() {
sftp -o IdentityFile=$Server <<-EOF  >$Processed_Logfile 2>&1
cd $Directory
lcd $2
get $1
echo $(date '+%Y/%m/%d %H:%M:%S') "     downloaded     " $1
rm $1
echo $(date '+%Y/%m/%d %H:%M:%S') "     deleted        " $1
!
EOF
}

# -----------------------------------------------------
# Loop job
# -----------------------------------------------------
while [ 1 ]
do
# -----------------------------------------------------
# Connect remote server get the directory list
# Save directory list in ftp_log.temp
# -----------------------------------------------------
sftp -o IdentityFile=$Server <<-EOF >$Logfile 2>&1
cd $Directory
ls -1
!
EOF
# -----------------------------------------------------
# Process line from list
# Get and remove file from remote server
# -----------------------------------------------------

while read p; do
  if [[ $p == *txt ]]
  then
    if [[ $p == AAA* ]]; then
      getdelete $p "/home/AAA/"
    elif [[ $p == BBB* ]]; then
      getdelete $p "/home/BBB/"
    elif [[ $p == CCC* ]]; then
      getdelete $p "/home/CCC/"
    elif [[ $p == DDD* ]]; then
      getdelete $p "/home/DDD/"
    elif [[ $p == EEE* ]]; then
      getdelete $p "/home/EEE/"
    fi
  fi
done < $Logfile

  #sleep in second
  sleep $Sleep
done >> $Script_Logfile 2>&1 &
4

Look at the following two threads.
Using a cron job is far more reliable than a long running script.
Also you are deleting the input file before you know that you can successfully read the copied file.
If the files are large, it is also possible that you will delete a file before it has been completely created.