I am using ncftpput in one of my c-shell script to upload files to a remote location. The source files keep coming continuosly and to upload each file ncftpput opens a new connection everytime. It means ncftp uploads the file1 than file2 than file3 .... There is gap 20-25 secs between each of the files and some times when its a peak time the backlog is getting too much high approx 100 source file
I have seen a project called filezilla, where you have the option to choose multiple files and its starts processing almost 2-3 files at a time. Its a great time saving tool, but I am unable to find that for AIX and there is no command line like ncftp :wall:.
Could any one among you may able to give me a hand please ?
Oh, you want to open dozens or hundreds of network connections simultaneously. Not usually a good idea. Too many connections can overwhelm a router, and the host you're uploading to might even consider it an attack.
Do you have ssh access to the machine? Can you upload an archive of some sort and unpack it there? Uploading one big file would avoid some overhead.
Can You pack the files? Simple tar with no compression should be sufficient. Than You would have to send one file instead of "many".
Besides, most ftp configurations doesn't allow more that 2-5 connections for the same user/IP/whatever.
No, I don't wat want to open tons of connections, but 2-3 threads would definately be ok. in my script a while loop continusly monitor the source forder. When a file arrives in the folder the program hand ot over to ncftpput. So any mechanism to open at least 2-3 thread would be ok any how.
Any comments please ?
Regards
---------- Post updated at 05:04 PM ---------- Previous update was at 12:33 PM ----------
GNU parallel is a shell tool for executing jobs in parallel using one
or more computers. A job is can be a single command or a small script
that has to be run for each of the lines in the input. The typical
input is a list of files, a list of hosts, a list of users, a list of
URLs, or a list of tables. A job can also be a command that reads from
a pipe. GNU parallel can then split the input and pipe it into commands
in parallel.
If you use xargs and tee today you will find GNU parallel very easy to
use as GNU parallel is written to have the same options as xargs. If
you write loops in shell, you will find GNU parallel may be able to
replace most of the loops and make them run faster by running several
jobs in parallel.
pdsh is a variant of the rsh(1) command. Unlike rsh(1), which runs
commands on a single remote host, pdsh can run multiple remote commands
in parallel. pdsh uses a "sliding window" (or fanout) of threads to
conserve resources on the initiating host while allowing some
connections to time out.
Thanks for your reply. I have actually knew both the programs, but they will open a new connection for every instance of the ncftp. I am actually looking for some solution in Unix like filezilla works on windows i.e. upload multiple files with a single connection. Please see if you can find something.
You can download a binary (or compile it yourself).
But I don't think FileZilla supports multithreaded operations (lftp seems to support parallel put/get though).
