I need to to write Bash script which will copy 5 files from local server (HP-UX) to Linux remote server into parallelized way, that is local server will have to open 5 copy istances (e.g. scp) to maximize transfer rate.
What do you think ?
@MAZZUGORO , if they're all going over a single link why do you think having N jobs would be (significantly) faster than a single job doing the transfer of all N files ?
(have you tried - taken timings etc ....)
there a any number of environmental factors to consider ... amongst
- how busy the link is
- buffer sizes on reading/writing
- how large the files being transferred are (this can influence 'speed' somewhat)
- I/O activity on the sending/receiving hosts
- application behaviour (does the application behave differently if files are received sequentially rather than spread across N transfers ...)
...
I'm sure other teammates will have comments/thoughts on this subject
Do you have rsync on HP-UX?
It can bundle all files to a stream that is via ssh protocol sent to the remote server and there split into the files again.
You can do it manually, using an archive to stdout and pipe it via ssh to the remote server and there unpack from stdin.
Example with tar
( cd srcdir && tar cf - file1 file2 ... ) | ssh remoteserver " cd destdir && tar xf - "
If bandwidth matters you can put gzip/gunzip in between:
( cd srcdir && tar cf - file1 file2 ... ) | gzip | ssh remoteserver " cd destdir && gunzip | tar xf - "
GNU tar has builtin (de-)compressors
( cd srcdir && tar cf - file1 file2 ... ) | gzip | ssh remoteserver " cd destdir && tar xzf - "
My examples copy the files in sequence, but most efficient, through a single connection.
1 Like