I'm writing shell scripts to handle incoming and outgoing automated sftp transfers between a local server and various remote servers belonging to different organizations.
I'm wondering if there are any recommended or "best practices" for signalling the end of a ftp file transmission (sending multiple files), such as sending an empty ".done" (e.g. "somefilename.done") or similar file, after all other files are sent. Or maybe sending one empty ".done" file per individual data file.
I've seen the ".done" approach here and elsewhere, and have one client who sends a ".eot" (end of transmission) file. I'm just wondering if there are any informal standards or practices for doing this.
The idea being that you don't want the party receiving files to start processing them until it's certain that all expected files have arrived and are complete.
Depending on the situation, I send to a filename that has a bang (!) as the last character. When the transfer is done, I rename the file to remove the bang. Same effect as creating a marker file, but less things in the directory to track. I also think it's easier to quickly see how many 'in progress' files exist. If I am really concerned about correctness, I send a 'done file' that contains the checksum of the data file and use the bang technique on both files. The receiving process then can verify that there were no errors in transmission and/or writing the file to disk.
Another technique that I use, when able, is to have the sender schedule the process on the remote host that will process the file after it believes the file has been sent successfully. No renaming or done files needed, and when I schedule the process, I can give it a checksum to verify that the file was received without error.
Making the client rename or move the file after transfer is best, I think. Less clutter, and nobody's going to mistake a blank file for the data. Moving to a completely different folder makes even less chance of error -- the file simply won't be there until it's complete, and it will appear instantly and whole. (as long as you moved it to a folder on the same filesystem, the file's contents don't need to be rewritten to move it.)