Have you tried searching for SFTP and script on this site? This sort of question has arisen numerous times before and so you should be able to find answers already.
Yes, I have tried to find something that i could use to solve this problem, but was unable to find here or posted on other forums. Even the problem appear to be trivial, it is not due validation part.
thanks for reply
After you have received the first batch, and sucessfully processed them, create a text file that looks like:
cd somewhere
delete file1.txt
delete file2.txt
...
mget *
quit
Then use this file as input to your next sftp session.
You will be deleting the files from the remote server just before retrieving the next batch.
Have done similar to this (recalled from memory) for ftp, I have not tried it for sftp:
# get files from sftp server
mkdir temp.$$
cd temp.$$
sftp << HERETEXT1
cd directory
bin
prompt
mget *
exit
HERETEXT1
# now delete files you have downloaded from sftp server
for FILE in *; do
sftp << HERETEXT2
cd directory
del ${FILE}
exit
HERETEXT2
done
I haven't tested this so do test it!
This assumes you have set up passwordless SSH for the account that is doing the sftp.
Something to consider.
The size of the files to be downloaded, and the frequency of collecting them.
Eventually you will download a file that is still in the process of being created.
You can overcome this, by creating two files for each download, the first the actual data file, and the second a one character file (same name different extension) to indicate the the file is complete.
Only process the received file if there is a corresponding second file.
Thanks so much for input!
yes, i have setup pwdless ssh and certs and its working just fine.
Will try to test yours suggestions!
I appreciate your help.
Thank you!
---------- Post updated at 03:29 PM ---------- Previous update was at 03:27 PM ----------
Jgt,
"Something to consider.
The size of the files to be downloaded, and the frequency of collecting them.
Eventually you will download a file that is still in the process of being created.
You can overcome this, by creating two files for each download, the first the actual data file, and the second a one character file (same name different extension) to indicate the the file is complete.
Only process the received file if there is a corresponding second file."
Somewhere in the creation of the FILE.TXT file on the remote server is a statement that either closes the file, or moves/copies it to the outbound directory.
Add a statement after that to echo the date, or job number, into a file called FILE.JOB and put this file in the outbound directory.