It takes 6 hrs for a 90 GB zip file that i am copying / transferring from serverA onto serverB.
scp user1@serverA:/opt/setup/cash.zip .
Output:
cash.zip 21% 19GB 4.7MB/s 4:11:46 ETA
uname -a
SunOS serverB 5.11 11.2 sun4v sparc sun4v
Can you please suggest if i could do something to speedup the file transfer ?
its based on the network bandwidth available between 2 locations. currently i see it transfers 4.7Mb/Sec.
where is server A and B is located ?
I dont know how far are they located as they are client servers. So i may need to inquire with them as i dont know how to figure it out myself.
I was looking more so from the command prospective if we can tweak it for faster transfer based on pre-conditions.
I will live with it if there is no way out.
If this is a regular transfer of a file or files that are updated (not completely re-written) you may have an improvement with rsync. This would examine the files at each end and only transmit the blocks that need changing so there should be less to transfer. Of course, if the target file does not exist, it will send the whole file, so you are stuck waiting on the network as above.
If you need to encrypt the connection (i.e. it's not a private network) then search for the string encrypt in the manual page and there are some examples there.
You can improve the network speed by paying for a faster link between the two sites, but that comes down to a business cost justification. I suppose with a complex network with many hops, it might be possible to improve things if a direct link could be set up, but this is entirely down to how the network is delivered. It might be possible to set up a VPN connection from one site to another over the public internet, but there are no guarantees that it will actually be any faster.
I hope that this helps,
Robin
What is the contents of the zip file?
Were a part of its files already transferred?
If yes, that is where rsync comes into play.
What are the operating systems at each end?
The option -C enable compression on the fly for scp command
@matrixmadhan: yes, but s/he is transferring .zip files that presumably are compressed, so the improvement by adding the -C option might not be noticeable.
I've found compression sending an already compressed file can be worse. If the file is large, then there is the cost of compression on-the-fly and the file can actually grow because there needs to be data added to describe how to decompress it, then the target has the cost of decompression.
Any of these can cause the transfer to slow down, mainly if the source or target are heavily CPU busy or memory is full and you cause paging.
Robin
I asked what the operating systems were in post#6.
If one of them is Windoze it would explain everything; period.
Windoze is hopeless at transferring very large files. It starts off fine but then gets slower and slower and slower, down to a crawl.
Are You Seeing Slow Windows 7 Network Performance?
And I'm not aware of any Windows version with this problem fixed.
@RudiC @rbatte1
It was only a general recommendation to use -C and that doesn't make sense to use it for files that are already compressed. Good catch!
Try not using encryption (if you can) as a first step of debug.
Something in the fashion of FTP or simple netcat [nc] utility from both sides (one listening other sending)
Try an NFS mount if you can, and compare speeds.
Couple of questions :
Are other clients (the servers you are copying to) exhibiting the same problem or just that one ?
If all the clients you are copying to experience slow copy -->
Check the netstat command (-s -i) and look for drops, errors and re-transmissions on the server side (the machine you are copying from).
How are disk response times and queues ? Are disks on the server side saturated perhaps ?
If disk read can not give you more 5MB/s no network will achieve more (unlikely, 5mb/s is quite low for todays 'standards', but check, iostat will be your friend here).
If only one (or couple of perhaps in same network VLAN), check out the clients with above commands.
Take output of commands before and after the action and compare.
The before is important, cause those counters are not restarted since boot probably, as a baseline.
Is the copy done over internet or intranet, are firewalls and/or packet inspection/throttling active on those networks ?
This is where netcat may can come into play, for instance, you can throttle 'per app' in firewall like ssh (port 22), nfs (port 2049) but netcat can use any port, test a couple on server/client, try using high port numbers.
And yes, i love netcat 