Copy from local to remote

Shell Programming and Scripting
1

Hi

I need a advice for writing simple bash script,

I have a file pod.txt which contains source location and remote location:

/mnt/infile/20141103/701_0001.png/remote/tmp/pk21730/p0330223723074.png
/mnt/infile/20141103/203_0001.png/remote/tmp/pk21731/p0330223723081.png

and I must copy data using above path from local to remote server:
(to better understand " scp /mnt/infile/20141020/701_0001.png /remote/tmp/pk21730/p0330223723074.png ")

I wrote this script:

#################
#/bin/bash

day=`date --date="1 day ago" +%Y%m%d`
Path1=/mnt/infile/$day
remote=node2@192.168.1.190:/tmp

awk -F '/' '{print "scp" " "  "'$Path1'" "/" $5 " " "'$remote'" "/" $8 "/" $9 }' pod.txt > tr.sh

chmod 755 tr.sh
client=`awk -F '/' '{print $8}'  pod.txt`

for file in $(echo $client);
do ssh node2@192.168.1.190 mkdir -p /tmp/$file;
done;

./tr.sh > tr.log 

##########

the script works fine but I'm afraid what will happen if the files to be copied will be about 10 000,

you have an idea for something with "for" ?

2

You certainly should not login to node2 10000 times. Create a shell script with all the mkdir s, login, and execute remotely. And, I'd create the entire file and directory structure locally, e.g. while read line ; do echo mv ${line/png\//png \/tmp\/}; done < pod.txt , and then copy that over with one single scp command.

3

Thank's RudiC

I changed the loop to create directories on a remote location
was:

for file in $(echo $client);
do ssh node2@192.168.1.190 mkdir -p /tmp/$file;
done;

is now:

ssh node2@192.168.1.190
while read line;
do mkdir -p /tmp/$line;
done; <<< $client

But I don't understand how to use one single scp command to copy all file from local server to remote server ? Can You give more details ?

4

scp offers the -r option to recursively copy directory trees. man scp :

So - create the tree locally, point scp to the tree root, and let it copy all the files including directories.
Do a test with small subset first. It will not work like a greased lightning, but not logging in 10000 time will make it significantly faster.

5

I think I can't use option -r because e.g file 1 on local server name "701_0001.png" but on remote server it must name "p0330223723074.png" (first of course I must create directory "pk21730")

6

That's why I proposed to create the entire new structure locally. If it's on the same file system, mv ing files will just modify the directory entries and is very fast. When that is done, copy the entire structure over.

1 Like
7

Ok , now it is clear to me , but to use this method I must calculated If I will have enough free disk space,
ThankS very much RudiC