File transfer from remote to local

Shell Programming and Scripting
1

Hi,
I came across the scenario, that I need to copy files from the remote server to my local. The files in the remote server are created by another job and its keep on generating the files in that remote folder.

We can't able to use SCP command and we're using SFTP to connect the server and mget to copy the files.

Once we copied the files we're giving rm command to wipe the directory of the remote server.

For example mget command moving 100 files and few more files are created in that folder bcoz of the job and while giving rm command it will remove the 100 files + newly created files from the remote server.

Is there any other best way to do this without losing the newly created files.

Thanks
Janarthan

2

Make a list of the files that you have processed on the receiving machine, and remove only those files from the sending machine just prior to the next mget.

3

How about rename ing the remote files to a temp directory, then get ing and thereafter deleting them? The newly created files will stay untouched in the original directory.

4

I think you are asking: 'how to stop deletion of files on a file server until after the file has been copied'

One way: rename each file right after you copy it over.
You can do this several ways example using your current pid

# create a special local directory
pid=$$
local_dir=./myfiles/${pid}
test -d  $local_dir || mkdir $local_dir
cd $local_dir
sftp someplace.com
username password
cd /path/to/files
mget files*
bye
find . -type f -name |
while read fname 
do
         sftp someplace.com <<EOF
         username password
         cd /path/to/files
         rename "$fname" "$pid"
         bye
EOF
        mv "$fname" /path/to/permanent_local_storage/directory
        rm "$fname"  # get rid of local duplicate. 
done
sftp someplace.com <<EOF
  username password
  cd /path/to/files
  rm "$pid"
bye
EOF

Your remote directory has one file left because you kept renaming every file you copied to a pid number. Remove it when you are done. This can be improved in lots of ways.

Note: the EOF that ends a heredoc in this example has to be LEFT i.e., column 1

EDIT: I see Rudi had the same idea.
It can float if the first one for the heredoc is <<-EOF

1 Like