I need to revise a transfer process. I have 11 "remote" servers all transfering 1 transfer file nightly. Now, this need to be more "automatic" in that the transfers need to "catch" all missed transfers up to the current date.
What can anyone suggest?
You didn't say how you are transferring the files.
You could delete files after they are transferred, thus leaving only those that were not.
11 servers each create a single transfer file of all "file" changes done to the local file that day. Earily morning transfers are done of the Internet and each server site receive the daily transfer. Then, "our" custom application read the transfer file(s) and process the update on the "other" server local copy. The transfer file is a standard text file.
Now, we need to figure out a method of letting the transfer detect the missed "daily" transfer and to get from the missed day.
No "central" location for all the transfer files would be considered, unless a very good valid reason.
Sorry, but could you explain it again? Do you have 11 servers that create 1 file, which then has to transferred to an additional server for processing? Or do you have one server that generates files for each of the 11 servers which then have to be transferred to them? Or do you have 11 servers that exchange those files with each other?
Also, how do you transfer these files now?
If what you are saying is that you are backing up changed files on the 11 servers to a backup server, you can just use rsync to backup the changes and if it missed anything it will catchup the next "sync".
But maybe I also dont fully understand what you are trying to do.
11 server each create 1 transfer file, containing all file changes done during the day.
11 servers each connect to the other servers so as to receive this transfer file and process the update to the "local" copy of the "other" branch files.
And my client has "refused" any recommendation for a centralized backup server.
Then rsync is probably the best bet for you. Create the files in a special directory, say /var/tmp/send. Loop through a list of all 11 servers and rsync to it, skipping if it's the current server. If a file is transferred correctly, remove it, otherwise rsync will take care of it on the next pass
Create a second directory, say /var/tmp/received, to which rsync sends the files, and from which they are then processed.
In order to avoid having rsync run in daemon mode, set up ssh on each server, which public key authentication, so that rsync can use that for transfer.