Copy file from different ports in parallel

Shell Programming and Scripting
1

Hello folks,

Can you please help me to solve the below concern.

I have a source server with 2 ports and have to copy the files from both the port to destination server simultaneously in my shell script.

How can I achieve that?

Source : x.x.x.x port -22
X.x.x.x port -2222

Destination: y.y.y.y

Requirements : want to do copy files from a specific location from both ports to destination server.

I have opened the password less ssh connection to both the ports, but want to perform scp parallel from destination.

Kindly guide

Regards,
Sadique Manzar

2

You can try something like the following:

ssh y.y.y.y
scp x.x.x.x:/path/to/files /path/to/destination &
scp -P 2222 x.x.x.x:/path/to/files /path/to/destination &
wait

The trailing ampersands make the commands run in the background, allowing for parallel operation of the two remote copy processes. The wait command makes the script wait for both processes to finish..

3

Hello Scrutinizer,

I need to call this from destination that is from y.y.y.y need to ssh x.x.x.x.

Kindly suggest

Regards,

4

Hi Sadique, isn't that what my suggestion is about?

5

Note carefully: There are limits to parallel copying. Routers and switches can be set to limit throughput - Ex: you can have 10GB but you are limited only 10% of 10GB. With no limits your copying could shutdown dozens of users on a network by causing their network access to creep along. This gets lots of folks mad, fast.

So parallel copying of a large number of files is not faster than copying, say, four files at a time. And FWIW ssh compresses transmissions by default. So turning on ssh compression, for example, on our large Solaris 10 boxes actually slowed down ssh (scp) file copies. Which always seemed paradoxical to me.

1 Like
6

Deae All,

My destination has open for both the port 22 and 2222.
So i need to copy the data from destination to my source.

I am using after getting help many here like this.

Here is the code:

while read -r line || [-n "$line"]
do
scp -C abc@x.x.x.x /path
scp -C -P 2222 abc@x.x.x.x /path
done < $destination_path

---------- Post updated at 08:10 AM ---------- Previous update was at 08:09 AM ----------

Please comment if any correction required.

7

Hi,

  • This is not simultaneously but sequentially (alternating) (see post #2)
  • You are reading variable line from a file whose name is contained in variable destination_path , which seems a bit strange
  • Yet this variable is not used within the loop, so you might as well leave the loop out if there is only one line in the file. Otherwise this does not make sense, since it would keep on overwriting the same file
  • [-n "$line"] is not valid code: there needs to be space around the square brackets.
8

Not only that, but it is also superfluous if it would work:

while read line ; do
    .....
done

will only run if something has been indeed read, because otherwise the read -shell-builtin would return an RC<>0 (and the loop would subsequently be left).

I hope this helps.

bakunin