I have a server that I have to ftp files off and they all start SGRD and are followed by 6 numbers.
SGRD000001
SGRD000002
SGRD000003
The script I have will run every 10 mins to pick up files as new ones will be coming in all the time and what I want to do is delete the files I have picked up from the ftp site as I don't want to keep downloading the same files.
Once I have downloaded the files I need to rename them. Before the renaming process I have done an ls SGRD* > Myfile on the files I have downloaded and then run a loop to put delete in front of the file names.
Is there anyway that I can run the commands within Myfile on the ftp server so that I just delete the files I downloaded ?
Another question I have is, is it possible to move the files that I have downloaded into a backup folder on the ftp server or do I have do download the files and upload them back to the ftp server ?
I tried rename in various different ways and it didn't work with multiple files.
# Assume the filenames live in a file called "infile"
echo "USER username password
cd /remote/path/to/files
`for f in cat infile; do
echo get $f
echo delete $f
done`
bye | ftp -n remoteplace
Because you are deleting the files and you cannot go back in time to get them if there is a problem:
I would consider making 2 connections - the first to get the files, check the files, then make a second connection to delete them.
Note: rather than putting passwords in the script consider using a .netrc file, and set the protections to 600. In general, ftp is insecure, and this helps only a little bit.
This whole operation would be much easier if you could do it from the ftp server itself. Do you have sysadmin rights on the ftp server?
Can you 'put' the files from there instead of 'get' the files? You can then move the files already 'put' to another directory.
Assuming you have the rights as the account you log in with ftp as, the del subcommand just needs to replace the get subcommand. If you have another way to verify that the transfers completed okay before you issue the deletes, then is there a problem in just looping for each file? There is no doubt a small overhead in putting a single delete command into ftp and then looping for each file, but it would make the code easier:-
while read file
do
ftp -inv $host <<-EOFTP >>ftp_log 2>&1
user $user $password
cd $target_directory
del $file
quit
EOFTP
done < Myfile
The credentials are included in the flow of commands to keep them off the ftp command line, else someone can see them with a simple ps, but like Jim says, it is better to use entries in a .netrc file. If you do, drop the -n flag or it will ignore the .netrc file. better still, if you can convert to sftp using ssh-keys to automate authentication it becomes easier again, but that may not be possible in your environment. It depends what you have for the client and the server and you need access on port 22 instead, so that might be one or more firewalls with a rule change.
If you just need to get the files and delete them, I would suggest two ftp commands per file so that you can check the transfer completes before you delete the source, something like:-
while read file
do
ftp -inv $host <<-EOFTP >>ftp_log.$file 2>&1
user $user $password
cd $target_directory
get $file
quit
EOFTP
# Verify here that the file arrived okay by reading the log ftp_log.$file
# If okay, then proceed with delete step
if [ whatever test you decide ]
then
ftp -inv $host <<-EOFTP >>ftp_log.$file 2>&1
user $user $password
cd $target_directory
del $file
quit
EOFTP
else
# Whatever error routine you decide
fi
done < Myfile
Note that the EOFTP to close the input to the ftp command must be either in the first column or tab indented. Space indented will cause it to be ignored and invalidate the input.
This is probably the server being unable to respond quickly enough. Just putting a small sleep in the loop outside the actual ftp should help. Of course if you sleep for 1 second but have 600 files that will take 20 minutes to complete if you sleep between every ftp and you open a second connection to do the deletion.
Does there seem to be a common limit to the number of connections when getting lots of small files? Perhaps you can have a sleep every 10 files so it doesn't hurt too much.