I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and there's no return code from the process id. Any ideas?
I thought about just using scp but scripting it is proving trickier than I thought and I'm getting errors on the for loop when I try to introduce variables into the ssh part:
E.g.
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
echo "attempting to copy file: " $BackupFile
scp -p $hostname:/db/remote_backup_area/$file /db/local_dump_area
# or use rsync
#rsync -av -e ssh oracle@${RHOST}:/db/remote_backup_area/${BackupFile} /db/local_dump_area"
done
exit
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
echo "attempting to copy file: " $BackupFile
scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area &
# or use rsync
done
wait
exit
which can swamp the connection. This sends 10 files at a go
cnt=1
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
echo "attempting to copy file: " $BackupFile
scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area &
cnt=$(( $cnt + 1 ))
if [ cnt -eq 10 ] then
wait
cnt=1
fi
done
wait
exit
Change the "10" to something that is reasonable
If you want some kind of TTY ouput you cannot do the above, you have to write to a file on the scp command line.
Hi Jim. Thanks for the reply. Did the above code work for you? I get an 'issue' in the first line of the for loop. It wont read the value of the $RHOST and $CCMD variables that are part of the loop condition .. so it does nothing unfortunately. If I hard code them in then it works .. I'm missing something from my code.
I find it hard to believe that even 1 rsync will not swamp your bandwidth as disk I/O should be WAY faster than the network.
If you want more speed try the -z (or --compress) rsync option and compress the data transfered.
Funny thing is I usually find that rsync swamps my network and even web service requests from/to the boxes concerned tend to timeout a lot. More often than not, I find myself using --bwlimit to slow things down!