Automate splitting of files , scp files as each split completes and combine files on target server

UNIX for Beginners Q & A
1

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B , once all the ten files have been received, i wll like to combine (cat them). How can i automate the whole process.
I want to save time and not wait for the completion of the splitting of all the ten 1000 GB files before I scp, I want to scp as each file completes.
Can anyone help with how to automate this process ? Thanks

2

You could "watch" the transfer directory on the receiving end and when a file arrives, you could start a process of waiting for the files to upload and auto cat them back together.

If you name your files in a clever way you can even make the required code quite simple, or you could transfer the meta data first and follow up with the files, where the meta data provides the information your "watch and combine" process needs.

1 Like
3

Hello malaika,

I would like to bring up a few things to consider:-

  • How do you know that the file is complete before you try to split it? Perhaps it gets transferred as filename.partial then renamed to filename when complete or there is a flag file such as filename.OKAY created in the next step
  • Why break up the file to send it and reassemble? Have you considered rsync? If there is an interruption it can restart to catch up. You might be able to set it up to transfer then remove the original. Trial this with small files though :wink:

Does this help/hinder/miss the point?

Kind regards,
Robin