Shell script to get all the files from FTP server

Hi Guru's,

      I am new to UNIX. my requirement is to log on to FTP server and get all the .txt files. i have developed one script by searching some forums but getting error and not able to fix them. pls see below code.
         
         ftp -i-n<<EOF
                open $FTP_HOST
                user $FTP_USER $FTP_PASS
                cd FTP_PATH
                ascii
                  for file in `ls *.txt 2>> $LOGFILE`
                    do
                      if [ ! -f $file ]; then
                        echo " `date` Files were not received " 
                      exit
                      else
                       lcd $LCK_IN_DIR
                       get $file
                      fi
                     done
                   bye
                   EOF

i am getting the below error when i run the script.

./ftp.sh[56]: syntax error at line 81 : `<<' unmatched

some how i have fixed that error. But script did not logon to FTP server automatically. when i run the script it is prompting to userid/password.

Appreciate your help

Thanks
Arun

the EOF must be at the very beginning of the line (no spaces)

But regardless, your approach is sadly not the right one. Use "mget *" to get all the files. Your for-loop in the FTP command just wont work.

A standard alternative is to use perl with the Net::FTP module (search these forums for that). Another approach is with "Expect". Again, search these forums.

#!/usr/bin/env ruby 
require 'net/ftp'
user="anonymous"
pass="password"
server="localhost"
Net::FTP.open(server) do |ftp|
    ftp.login(user,pass)
    ftp.nlst("*.txt").each do |file|
       ftp.getbinaryfile(file,file)
       ftp.close
    end
end


1 Like

Sexy. I like that. How do install ruby? :wink:

Thanks KURUMI,

I forgot to mention couple of points here.
I am using HP UNIX and i need download all the files in ASCII mode. will the above script work?
Thx
Arun

hi otheus, please see Download Ruby
Depending on your distribution, there might be a package manager that makes installation easy.

---------- Post updated at 08:52 AM ---------- Previous update was at 08:47 AM ----------

sorry, HP UNIX does have Ruby by default. So unless you install it, you can try Perl instead. But judging from what you already have, you can simply use

mget *.txt  

Hi Kurumi,

I think ruby is not installed and I am getting error. Could you pls provide sample code using Perl. I have tried manually mget *.txt in FTP server. looks like mget is not working in FTP. Will it work with perl?

Thanks for your help again.

Thanks
Arun

"looks like"? What is it actually doing? What does it say? Does it recognize the command at all? If you won't let us help you troubleshoot, there's not much point throwing code at you either; when it "doesn't work" too, we'll still be pulling teeth to try and find out why.

You could try:

use Net::FTP;

my @serverip=qw/serverip1 serverip2/;
$fileextension="*.txt";

foreach my $serverip(@serverip){
my $ftp = Net::FTP->new($serverip,Debug=>0)
or die "can't connect";

$ftp->login("username","password")
or die "cannot login";
$ftp->cwd("/the ftp directory")
or die "cannot change directory";

foreach my $file($ftp->ls($fileextension)){
$ftp->get($file) or die "get failed", $ftp->message;
printf("done for $file\n");
}
$ftp->quit;
}

hi Phil,

I have used your code accordingly. but getting the error at fist line itself like use: not found,my: not found,what i need pass in serverip1 serverip2? is that source IP and target IP? Also where can i pass the local server path to copy the file? where i can mention about ASCII mode?
Thanks
Arun

Try actually using perl to run that script instead of running it in your shell.

A corrected version of post #1 (in Shell) confined to those commands which you can use in ftp. Note that ftp is not shell and ftp has its own syntax (and we have only used ftp syntax within the shell here document delimited by the string "EOF").

Obviously you will need to set values in the first 5 variables to match your FTP server account and your local directory.

FTP_HOST="ftp_server_hostname"
FTP_USER="remote_username"
FTP_PASS="remote_password"
FTP_PATH="/remote_pathname"        # Remote directory
LCK_IN_DIR="/local_directory"     # Local directory
#
ftp -i -n <<EOF
open ${FTP_HOST}
user ${FTP_USER} ${FTP_PASS}
cd ${FTP_PATH}
ascii
lcd ${LCK_IN_DIR}
mget \*.txt
quit
EOF

Thanks Mythyl,

I have modified you code accordingly. when i execute the code i am getting error at mget \*.txt:

Local directory now /tmp
*.txt: No such file or directory

I have once .txt file in remote location. I am using HP UNIX.

Appreciate your inputs.

Try the ftp commands manually, making sure that you have the remote directory name correct.
Could also try "dir \.txt" instead of "mget \.txt" for testing purposes.

Methyl,
Looks like FTP system is not allowing wild characters. Here is the output for dir \*.txt

ftp> dir \*.txt
200 PORT command successful
150 Opening ASCII mode data connection for file list
226 Transfer complete
ftp>

but when i use dir or ls -l here is the output.

ftp> ls -l
200 PORT command successful
150 Opening ASCII mode data connection for file list
-rw-r--r--   1 ftp      ftp           656 Sep 30 19:00 R1_1380933_DATA_09292010135215.txt
226 Transfer complete
ftp>

any suggestions?

Thanks
Arun

Typing at the command prompt is different from preparing commands in a Shell script.

When typing directly at a ftp command prompt the Shell does not process what you type.

From a ftp command prompt you don't need the backslash character because we are not protecting the asterisk from the Shell. So, when testing without using a Shell script you need to omit the backslash.

I have tried without backslash from ftp command prompt. but the same results.

ftp> dir *
200 PORT command successful
150 Opening ASCII mode data connection for file list
226 Transfer complete
ftp>

ftp> dir *.*
200 PORT command successful
150 Opening ASCII mode data connection for file list
226 Transfer complete
ftp>

but file is displayed when i do ls -l

ftp> ls -l
200 PORT command successful
150 Opening ASCII mode data connection for file list
-rw-r--r--   1 ftp      ftp           656 Sep 30 19:00 R1_1380933_DATA_09292010135215.txt
226 Transfer complete

Thanks
Arun

It think it should be mget *.txt instead of mget \*.txt , no?
The asterisk is protected from the shell since it is inside a here document (where there is only parameter expansion, command substitution, and arithmetic expansion)

Are both computers HP-UX ? If so, what version(s)?

We seem to have a very strange version of ftp here which does not react correctly to globbing "*".

What is the output from the ftp "status" command?
Reference your last post, what is the output from ftp "dir" command with no parameters.?

wget has an option for recursivity i used it to suck up a hole website !!! thousands of photos lol