Every day i ftp tar.gz a file from the production server to a back up machine.. This task creates way to much traffic on the network at the end of the day and puts and undo load on the production machine during operation hours. i would like to create a script that would automatically fire off the ftp at like 2:00 am so that the transfer occurs when there is little or no load on the network. if anyone has suggestions on how to accomplish this, PLEASE let me know...
thanks..
--e0
You can set up a ".netrc" file for the
user doing the transfer then set up a cron
job for that userid to run a script to actually
do the FTP transfer.
Check the man pages for ftp(1) and netrc(5)
I need to do a lot of automated ftp jobs myself. I have tried several versions of this and have finally settled on a style of script that I like. I found that using a .netrc file to automate the logging-in process kept painting me into a corner because different scripts needed to sign in as different users. So I avoid .netrc and force the script to sign in. I don't like to allocate pty's unless I really am forced into it, so I also avoid pty based tools like expect. I really like ksh so that was my tool of choice. And I like the co-process concept because it makes feeding commands into the ftp process so easy with "print -p". The only problem is that the co-process manipulates standard-out so as to make it available to "read -p". And it's too hard to know how many "read -p" I will need. So I send the output to a different file descriptor. Putting it all together:
#! /usr/bin/ksh
HOST=remote.host.name
USER=whoever
PASSWD=whatever
exec 4>&1
ftp -nv >&4 2>&4 |&
print -p open $HOST
print -p user $USER $PASSWD
print -p cd directory
print -p binary
print -p put tar.gz
print -p bye
wait
exit 0
That script will tranfer the file and the output of the script will be the output from the ftp job itself. Put the script into cron and save the output so you can look at it the next morning.
hmm... write a script on the server that is launched daily using the cron (man crontab)... have that script ftp to your workstation or wherever and grab the file. The automated FTP will probably require a .netrc file in your user directory or wherever you ftp to to get the file.
However, if you are having to tar the file manually every day, write a script on the workstation side to do that for you a few minutes before the server script executes... once again, utilizing the crontab.
Good luck and let me know how it goes.
Strange... those other replies both popped up while I was writing... ummm... take their advice. 
Perderabo... can you explain what that does exactly?
That is the file descriptor manipulation. Recall that fd 0 is standard-in, fd 1 is standard-out, and fd 2 is standard-error. The line "exec 4&>1" opens fd 4 and assigns it to whatever fd 1 was assigned to. As you will see, I am sorta "saving a copy of fd 1 in fd 4".
The line "ftp -nv >&4 2>&1 |&" is a little harder.
The "|&" turns the process into a co-process that allows subsequent "print -p" statements to send lines to the co-process' standard-in and "read -p" to read from its standard-out. So ksh forks a copy of itself and fiddles with the fd's 0 and 1 until this it set-up. But it leaves the rest of the fd's alone.
Then it encounters ">&4" which causes it to set the ftp process' standard out to whatever 4 is. Well since 4 is a copy of 1 before the co-process, we are back to writing to the original shell's standard out. Lastly, the 2>&4 does the same thing for standard error. I could've used "2>&1" at this point for the same effect.
This is hard to explain, but I hope this helps.
thanks for all of the help.. i guess i should have mentioned that i am running sco openserver.. i dont know if it matters or not, but the more information the better.... anyway, it looks like the best way to do this is to set up a script and use crontab..
manually, i was using the get command from the back up machine, but if its all the same, put from the production machine will work fine.
now, its time to get down to the creation of the script and setting up cron to run it...
my unix background is about three weeks, but i have been an (i apologize in advanced) NT admin for over a year, so please forgive my ignorance.
anyway, if any of you would like to assist me in setting this up i would be very gratefull... This website, Unix: the complete reference, and sco unix in a nutshell are the only resources i have so far, so i am relying on you all a little...
thanks again...
e0--
anyone else remember using Vi for the first time??? is this our only option???
e0-
Vi hater
Well... I first started using it 17 years ago
and all I can remember is that it was 1000 times
better than "ed" 
You do have emacs as another option but I
don't think you'll like that much better.
If you have X, you should be able to find
a X text editor (perhaps pretty basic).
The thing about vi is that you can pretty much
find it on any **IX system. Anything else and
all bets are off.
I'm sure you could find a substitute on the net
somwhere but I've never looked for one myself.
Sorry about that 
If you're using Linux, or have pine installed, look for pico. In any case, I too would recommend vi. Again, if you on a Linux system, and vim is installed (Vi IMproved), try the command vimtutor.
HTH
First off, thanks for explaining that Perderabo! I thought that was all dealing with fds, but I wasn't sure. I love this forum...
As far as editors, vi is probably the most powerful, but one of the hardest to get used to... I myself used to be a vi hater...
But there are several text editors available... kwrite (kde), pico, joe... the possibilties are endless. Several of them will color code scripts as you write them too. It's really quite fun! ... hmm... does that make me a nerd?
Anyways, don't forget crontab -e to edit your crontab. It will use whatever you have the $EDITOR set to.
In case you didn't know, here's the syntax for crontab:
MIN HOUR DAY MONTH DAYOFWEEK COMMAND
ie. 5 2 * * * /home/username/bin/script
That will run your script every day, at 5 after 2 in the morning.
Good luck and don't be afraid to ask questions.
thanks again.. this is what i have for the script (after wrestling with Vi for about half an hour.)
text nested in /* */ is not actually in the script, these are my questions to the forum...
#!/usr/bin/ksh
/* is this neccessary, or can i run it in the default shell?*/
HOST=10.0.0.3
USER=root
PASSWD=*****
/* what exactly do these two lines do? */
exec4>&1
ftp -nv >&4 2>&4 |&
print -p open $HOST
print -p user $USER $PASSWD
print -p cd /
print -p cd /u/home
print -p lc ..
print -p binary
print -p put rwc8.tar.gz
wait
/* does this wait until the ftp completes, or is it expecting a number? */
exit 0
i pretty much copied exactly what perderabo posted a few replies up...
two questions:
1) is it ok to run this job as root or should i create an ftp user for this task??? i really dont like the fact that the root password is in clear text in this script.. if someone stumbled across it and opened it they would have the root pw..
2) i would like the daemon to send me a mail with the results job.. the text that is spit out when the ftp session completes would be fine, just so i know how much data was transferred.
thanks again..
e0-
The script must run as ksh since you are
using ksh built-in's (print) and capabilities.
I would set up a new user to run this as.
You may have problems logging in as root
via ftp on the remote unless you have configured
the remote to allow this (I try to avoid using
root where ever possible).
Check out the "tee" command to set up a
" | mail someuser@somehost.com"
The "wait" will wait for the child(s) to
complete.
Hi:
I tried your script .. somehow it doesnt work with SFTP it still asks for password......
#!/usr/bin/ksh
exec 4>&1
sftp user@host >&4 2>&4 |&
print -p cd directory
print -p put test
wait
exit 0
~
Any suggestions?
Hi:
I tried your script .. somehow it doesnt work with SFTP it still asks for password......
#!/usr/bin/ksh
exec 4>&1
sftp user@host >&4 2>&4 |&
print -p cd outgoing
print -p put test1
wait
exit 0
~
Any suggestions?
I would use Perl. Any version newer that 5.0x has the necessary Net::FTP module.
\#!/usr/bin/perl
use Net::FTP;
$ftp = Net::FTP->new\("MACHINE_NAME", Debug => 0\)
or die "Cannot connect to some.host.name: $@";
$ftp->login\("USER_NAME",'PASSWORD'\)
or die "Cannot login ", $ftp->message;
$ftp->cwd\("/pub"\)
or die "Cannot change working directory ", $ftp->message;
$ftp->get\("that_file.tar.gz"\)
or die "get failed ", $ftp->message;
$ftp->quit;
You could then stuff this into the user cron. Hard coding the username and password does present obvious security issues. If you're interested in Perl, check out cpan.org and perlmonks.org
Does this option allow for getting a specific file? What I am looking for is to ftp the log from the previous day that is in YYYYMMDD.log format. I don't wnat to use the mget and keep transfering the older logs as well.
Thx for any updates.
Scott
Thank you very much for the explanation and example of this script. I had to install korn shell for the command interpreter, but at 330KB I think I can spare the diskspace.
Here is a link to my application of this.
[NASLite server](http://web.ripnet.com/~rdeschen/NASLite server setup - backup server.html)
Rick D.
Brockville, Canada
ups sorry its not this hall
does anyone have a C shell version of this
#! /usr/bin/csh -f
#*************************************************
#! /usr/bin/ksh
HOST=remote.host.name
USER=whoever
PASSWD=whatever
exec 4>&1
ftp -nv >&4 2>&4 |&
print -p open $HOST
print -p user $USER $PASSWD
print -p cd directory
print -p binary
print -p put tar.gz
wait
exit 0