Rsync to an external list of URLs

Shell Programming and Scripting
1

I'm going to have a text file formatted something like this:

some_name http://www.someurl.com/
another_name http://www.anotherurl.com/
third_name http://www.thirdurl.com/

I need to write a script that can rsync from a file path I'll set, to each URL in the list.

Any ideas?

2

Post what have you tried so far and where exactly are you stuck?

Please read Simple rules of the UNIX.COM forums: before posting, especially 5 and 6.

3

Re: the rules; it's not for homework, and I have searched but didn't find anything relating to this specific idea.

Frankly, I don't know where to start. I've been looking at Awk and at Bash itself, so far.

4

First step is to read each address for file:

while read name URL
do
   # do something with that $URL but I don't belive you can rsync from url
done < url_file

.. and next you have to visit rsync and see what you can or you cannot do.

5

OK, so I'm able to loop through the external file and read a line into a variable. (I'm working with Mac OS X.)

SRCURL="~/Desktop/listrsync/source/";
while read URL
do
	echo "$URL";
	echo "rsync -avz --dry-run $SRCURL $URL";
	rsync -avz --dry-run $SRCURL $URL;
done < ~/Desktop/listrsync/targets.txt

Within the targets.txt file, I have:

~/Desktop/listrsync/a/
~/Desktop/listrsync/b/
~/Desktop/listrsync/c/

When I run the script, it produces the output:

~/Desktop/listrsync/source/
~/Desktop/listrsync/a/
rsync -avz --dry-run ~/Desktop/listrsync/source/ ~/Desktop/listrsync/a/
~/Desktop/listrsync/b/
rsync -avz --dry-run ~/Desktop/listrsync/source/ ~/Desktop/listrsync/b/
~/Desktop/listrsync/c/
rsync -avz --dry-run ~/Desktop/listrsync/source/ ~/Desktop/listrsync/c/

As you'd said, the rsync command itself doesn't work when run from within the loop. I'm wondering, since I can get validly-formatted rsync commands from the output, is there a way to simply run them?

Maybe through piping?

Or a temp file?

6

I just tried directing the output (valid rsync commands) to a temp file, and running the temp file as a script; and it worked. Is there a better way?

7

Is there a way to get the line number from the while loop, or should I start a counter for that?

8

Would this work? I'm trying to:

  1. Read first line into a variable.
  2. Loop from second line to EOF.
    3. Write rsync command with variable as source and current line as target.
  3. Pipe results of loop to sh to run as script.
url_list='~/pathto/file.txt'
current_source=$(awk 'NR==1'  "$url_list")
x=2
for (( $url_list-1 )) in $url_list
print "rsync -avz $(awk "NR==$x" "$url_list")" $current_source
$x++ | 
sh
9

Should work in the loop, try to set the full path to rsync.