FTP Script Help!!

gkelly1117 · October 31, 2014, 5:28pm

I am trying to write a script that allows me to check a directory for files that i will sftp to another server.

When the file is created it creates a .ready file that is supposed to signal a file is fully created and ready to sftp. for example a file named 10312014.13.07.40.27711113 would have trailing 10312014.13.07.40.27711113.ready file as well.

i dont know how to properly check for 10312014.13.07.40.27711113.ready and if it's there sftp 10312014.13.07.40.27711113.

below is what I have so far, it's just a shell of an idea, I'm stuck and any help would be greatly appreciated.

Thanks!

#!/bin/bash
source /opt/foobar/$1/config/usrconfig.sh
source  /opt/foobar/$ENV/config/$2

### foobar environment variables ###
export FTPHOST="sftpserv.foobar.com"                                    # server to ftp to
export FTPUSR="usmssh"                                              # ftp user id
export FTPPASS=""                                      # ftp user password
export PATTERN=                                      # pattern for file name to ftp
export USING_MARKER=no                                          # using a marker file???
#export TDEMARKER="X"                                    # pattern for marker file if no files to ftp
export FTPDIR="prod/TDE/in/"  # directory to ftp to on ftp server
export DAEMONLOG=/opt/foobar/$ENV/log/TDEdeamon.log
export DAEMONSLEEP=10
export CURRDATE=`date +%Y%m%d`
export READDIR=/opt/foobar/$ENV/data/TDE/output
export TEMP=/opt/foobar/$ENV/tmp
export TDE_BACKUP=$APP_ROOT/data/TDE/backup/

cd $READDIR
if [ -f  $READDIR ]
   then
     echo " $CURRDATE files found to send!"
   else
     echo "no files found in $READDIR to send!"
     exit
fi
sleep 2


for i in `ls $READDIR`
 do
        cp $i $TDE_BACKUP
        echo $i


        if [ -f $i.ready ]
        then
         rm *.ready

			/usr/bin/sftp $FTPUSR@$FTPHOST << E_O_F
			cd $FTPDIR
			put  $i
			ls -l $PATTERN*
			quit
E_O_F
        else
        echo "No files are ready"
        fi
done

.

Corona688 · October 31, 2014, 5:47pm

for I in `ls *` is a useless use of backticks. for I in * does the exact same thing with one fewer command and no unintended side-effects.

You can upload as the name you want with "put localfilename remotefilename".

You can feed the list of files into sftp by using a pipe, avoiding running sftp 537 times for 537 files.

#!/bin/sh

(
        cd $READDIR

        echo "cd $FTPDIR"

        for FILE in *.ready
        do
                [ -e "$FILE" ] || break # Avoids error
                echo "put ${FILE} $FTPDIR/${FILE/.ready/}"
        done

        echo "ls -l $PATTERN"
        echo "quit"
) | /usr/bin/sftp $FTPUSR@$FTPHOST

This just uploads the file, I wasn't quite sure what else you wanted done with them.

RudiC · November 1, 2014, 8:10am

You'll need to slightly modify Corona688's proposal for this:

        for FR in *.ready 
          do FILE="${FILE/.ready/}" 
             [ -e "$FILE" ] || break # Avoids error
             echo "put ${FILE} $FTPDIR/${FILE}"

This will get all "*.ready" filenames, remove the ".ready" part, and sftp the actual file as desired. Is the dot at the end part of the filename or a full stop? If the former, adapt the expansion pattern.

Don_Cragun · November 1, 2014, 3:15pm

rudic:

You'll need to slightly modify Corona688's proposal for this:
   for FR in *.ready 
   do FILE="${FILE/.ready/}" 
   [ -e "$FILE" ] || break # Avoids error
   echo "put ${FILE} $FTPDIR/${FILE}" 
This will get all "*.ready" filenames, remove the ".ready" part, and sftp the actual file as desired. Is the dot at the end part of the filename or a full stop? If the former, adapt the expansion pattern.

Almost. You used FILE where you needed FR:

        for FR in *.ready 
          do FILE="${FR%.ready}" 
             [ -e "$FILE" ] || break # Avoids error
             echo "put ${FILE} $FTPDIR/${FILE}"

I also switched from ${var/expr/rep} to ${var%pattern} since the latter is in the standards and is accepted by more shells.

Depending on how solid your file creator is, you might also want to change the break to a continue . Or to do what I think Corona688 was trying to do, change that line to:

             [ -e "$FR" ] || continue # Avoids error

which catches the case where there are no matching files and FR will be set to the literal string *.ready . (But, there isn't likely to be a flle named * either, so this probably doesn't matter.)

gkelly1117 · November 3, 2014, 11:38am

Thanks so much guys for your help! your input helped me get over my first hurdle, however another one arose that I am having issues with, and I am sure it has to do with how I coded this thing up.

I need to send multiple files, not just one. I thought I could send multiple like this, but I cant, so I played with creating a batch file, but I am clearly not doing this right.

Please see below code.

What am I doing wrong now? Thanks in advanced!

#!/bin/bash -x
source /opt/foobar/$1/config/usrconfig.sh

### foobar environment variables ###
export FTPHOST="sftpserv.foobar.com"                                    # server to ftp to
export FTPUSR="foouser"                                              # ftp user id
export FTPPASS="foo123"                                      # ftp user password
export PATTERN=                                      # pattern for file name to ftp
export USING_MARKER=no                                          # using a marker file???
#export TDEMARKER="X"                                    # pattern for marker file if no files to ftp
export FTPDIR="/home/foo"  # directory to ftp to on ftp server
export DAEMONLOG=/home/foobaruser/TDEdeamon.log
export DAEMONSLEEP=10
export CURRDATE=`date +%d-%m-%Y-%H-%M`
export READDIR=/home/foobaruser/TDE
#export TEMP=/opt/foobar/$ENV/tmp
export TDE_BACKUP=/home/foobaruser/TDE/backup

cd $READDIR
if [ -a  $READDIR ]
   then
     echo " $CURRDATE files found to send in $READDIR!" >>$DAEMONLOG 2>&1
   else
     echo "no files found in $READDIR to send!" >>$DAEMONLOG 2>&1
     exit
fi
sleep 2

touch tde_batch
for i in *.ready
 do {
 FILE="${i%.ready}"
 [ -e "$FILE" ] || continue
  echo " Going to put ${FILE} in $FTPDIR directory" >>$DAEMONLOG 2>&1
  echo "put ${FILE}" >> tde_batch
  }
  echo "quit" >> tde_batch
  
  sftp -b tde_batch $FTPUSR@$FTPHOST  
  
        cp $i $TDE_BACKUP
            cd $TDE_BACKUP
            rm *.ready
done

Corona688 · November 3, 2014, 11:53am

Moderator comments were removed during original forum migration.

rbatte1 · November 3, 2014, 11:59am

What errors do you get?

You might be safer to remove a single .ready files within in the loop so if any more arrive whilst you are processing, they don't get the flag removed.

I don't see why you have the curly brackets/braces in the loop. Can you explain? You are sending a single file each time. Is this intended? I'd be happy with that, but you complicate it with the wildcard rm *.ready

Robin

gkelly1117 · November 4, 2014, 10:31am

Sorry about that, I will post proper errors that can be used

---------- Post updated at 12:14 PM ---------- Previous update was at 12:13 PM ----------

No I want to send multiple files if possible. as for the curly braces, I thought they would help my problem. No real reason other than that.

---------- Post updated at 12:19 PM ---------- Previous update was at 12:14 PM ----------

I ran the script with a -x below is the output.

It looks like it's running fine until it finds the first file, it echo's the file, writes it to the batch file, then echoes the quit to the batch file, I want to traverse, echo every single file into the batch file first, then echo the quit, then let the batch file sftp.

My logic is clearly wrong, some proper directive would be appreciated!

Thanks! I hope this is better information to help with. Also I see the permission denied error, but I assume that batch needs ssh keys shared and not a password, in the production env, the keys are already shared and that wont be an issue

+ source /opt/foobar/or_foo/config/usrconfig.sh
+ export FTPHOST=foobar01.foobar.com
+ FTPHOST=foobar01.foobar.com
+ export FTPUSR=foouser
+ FTPUSR=foouser
+ export FTPPASS=fooy123
+ FTPPASS=fooy123
+ export PATTERN=
+ PATTERN=
+ export USING_MARKER=no
+ USING_MARKER=no
+ export FTPDIR=/home/foouser
+ FTPDIR=/home/foouser
+ export DAEMONLOG=/home/foobaruser/TDEdeamon.log
+ DAEMONLOG=/home/foobaruser/TDEdeamon.log
+ export DAEMONSLEEP=10
+ DAEMONSLEEP=10
++ date +%d-%m-%Y-%H-%M
+ export CURRDATE=03-11-2014-10-07
+ CURRDATE=03-11-2014-10-07
+ export READDIR=/home/foobaruser/TDE
+ READDIR=/home/foobaruser/TDE
+ export TDE_BACKUP=/home/foobaruser/TDE/backup
+ TDE_BACKUP=/home/foobaruser/TDE/backup
+ cd /home/foobaruser/TDE
+ '[' -a /home/foobaruser/TDE ']'
+ echo ' 03-11-2014-10-07 files found to send in /home/foobaruser/TDE!'
+ sleep 2
+ touch tde_batch
+ for i in '*.ready'
+ FILE=10312014.10.36.51.52667612
+ '[' -e 10312014.10.36.51.52667612 ']'
+ echo ' Going to put 10312014.10.36.51.52667612 in /home/foouser directory'
+ echo 'put 10312014.10.36.51.52667612'
+ echo quit
+ sftp -b tde_batch foouser@foobar01.foobar.com
Permission denied (gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive).
Couldn't read packet: Connection reset by peer
+ cp 10312014.10.36.51.52667612.ready /home/foobaruser/TDE/backup
+ cd /home/foobaruser/TDE/backup
+ rm 10312014.10.36.51.52667612.ready
+ for i in '*.ready'
+ FILE=10312014.11.15.41.64934033
+ '[' -e 10312014.11.15.41.64934033 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.11.22.53.52444830
+ '[' -e 10312014.11.22.53.52444830 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.34.26.49837220
+ '[' -e 10312014.12.34.26.49837220 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.36.16.5815654
+ '[' -e 10312014.12.36.16.5815654 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.47.03.24215656
+ '[' -e 10312014.12.47.03.24215656 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.12.49.21.40757399
+ '[' -e 10312014.12.49.21.40757399 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.01.41.84187558
+ '[' -e 10312014.13.01.41.84187558 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.07.18.23453400
+ '[' -e 10312014.13.07.18.23453400 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.07.40.27711113
+ '[' -e 10312014.13.07.40.27711113 ']'
+ continue
+ for i in '*.ready'
+ FILE=10312014.13.26.27.08714191
+ '[' -e 10312014.13.26.27.08714191 ']'
+ continue



[foobaruser@foobar02 TDE]$ cat tde_batch
put 10312014.10.36.51.52667612
quit

---------- Post updated 11-04-14 at 10:31 AM ---------- Previous update was 11-03-14 at 12:19 PM ----------

I think I need gidance on how to properly loop the traversing of files being put into the batch file before echoing quit

rbatte1 · November 4, 2014, 12:06pm

It looks like the logic could do what you are after. What do you get if you just run the following on the command line?:-

sftp foouser@foobar01.foobar.com

It might just be that the authentication is missing or incomplete.

If you get a prompt asking if you want to accept a fingerprint, then accepting it will store the fingerprint (effectively the remote server public key) in the know_hosts file so it will not prompt again. That has been a problem I've seen before sorted by a manual test first.

Thanks, in advance,
Robin

gkelly1117 · November 4, 2014, 4:50pm

I got it to work finally.

Below is finished product.

Thanks for all involved!

#!/bin/bash -x
source /opt/foobar/$1/config/usrconfig.sh

### foobar environment variables ###
export FTPHOST="sftpserv.foobar.com"                                    # server to ftp to
export FTPUSR="foouser"                                              # ftp user id
export FTPPASS="foo123"                                      # ftp user password
export PATTERN=                                      # pattern for file name to ftp
export USING_MARKER=no                                          # using a marker file???
#export TDEMARKER="X"                                    # pattern for marker file if no files to ftp
export FTPDIR="/home/foo"  # directory to ftp to on ftp server
export DAEMONLOG=/home/foobaruser/TDEdeamon.log
export DAEMONSLEEP=10
export CURRDATE=`date +%d-%m-%Y-%H-%M`
export READDIR=/home/foobaruser/TDE
#export TEMP=/opt/foobar/$ENV/tmp
export TDE_BACKUP=/home/foobaruser/TDE/backup

cd $READDIR
if [ -a  $READDIR ]
   then
     echo " $CURRDATE files found to send in $READDIR!" >>$DAEMONLOG 2>&1
   else
     echo "no files found in $READDIR to send!" >>$DAEMONLOG 2>&1
     exit
fi
sleep 2

touch tde_batch
cd $READDIR
find $READDIR -type f -name '*.ready' | while read file
do
  real_file="${file%.ready}"
  echo "put $real_file $FTPDIR" >> tde_batch
  cp $real_file $TDE_BACKUP
done
echo "quit" >> tde_batch
sftp -b tde_batch "$FTPUSR@$FTPHOST"
cp tde_batch $TDE_BACKUP/tde_batch.$CURRDATE
rm tde_batch  
rm *
exit 0

Chubler_XL · November 4, 2014, 5:22pm

Be careful of the rm * at the end of your script.

If any files are created while the sftp transfer is running they will be removed without being sent to sftpserv.foobar.com.

Aia · November 4, 2014, 6:49pm

I would like to bring to your attention the following combination of commands in red, which has the potential of spread disaster.

The command cd will silently change directory to your $HOME if it can not use $READDIR to change. Once's in there I don't know the mayhem that it would do, but for sure you can kiss goodbye to your data in $HOME.

Some ways to minimize the risk is to use absolute paths and if you cd check $PWD or pwd, and compare with where you should be.

gkelly1117 · November 4, 2014, 10:29pm

The rm * bothers me as well. Any suggestions would be greatly appreciated. I just want to delete the files after I have copied them to a back up directory and SFTP'd them over

Chubler_XL · November 4, 2014, 10:40pm

How about:

find $READDIR -type f -name '*.ready' | while read file
do
  real_file="${file%.ready}"
  echo "put $real_file $FTPDIR" >> tde_batch
  echo "!rm $real_file" >> tde_batch
  cp $real_file $TDE_BACKUP
done

gkelly1117 · November 4, 2014, 10:49pm

chubler_xl:

How about:

find $READDIR -type f -name '*.ready' | while read file
do
  real_file="${file%.ready}"
  echo "put $real_file $FTPDIR" >> tde_batch
  echo "!rm $real_file" >> tde_batch
  cp $real_file $TDE_BACKUP
done

You're awesome thanks a lot!!!

Chubler_XL · November 4, 2014, 10:52pm

The other advantage of this technique is that if the sftp fails on one of the transfers for any reason while in batchmode all processing of the batchfile stops at that point so there is much less danger of deleting a file that wasn't successfully sent .

Don_Cragun · November 4, 2014, 11:40pm

Aia,
No. No. No!

The only three ways for the command:

cd $READDIR

to silently change directory to a user's home directory:

READDIR is unset or set to a blank string,
READDIR expands to an absolute pathname naming the user's home directory, or
READDIR expands to a relative pathname that moves up or down within the file hierarchy from the current working directory to the user's home directory.

And, since gkelly1117's code includes:

export READDIR=/home/foobaruser/TDE

four lines before using $READDIR , we know that none of the above apply. If, however, the invoking user does not have permission to search /home or /home/foobaruser , or one ore more components of /home/foobaruser/TDE does not exist; then the current working directory will not be changed and a diagnostic message will be printed.

If you want to know if and invocation of cd succeeded, the proper thing to do is to check its exit status.

_____________________________________________________________
gkelly1117,
I don't understand what you're trying to do with the following code:

cd $READDIR
if [ -a  $READDIR ]
   then
     echo " $CURRDATE files found to send in $READDIR!" >>$DAEMONLOG 2>&1
   else
     echo "no files found in $READDIR to send!" >>$DAEMONLOG 2>&1
     exit
fi

Most programmers would check to see if a directory exists before trying to cd to it, or would check the exit status of the cd command. Instead of that, you blindly cd , ignore the exit status, and then check to see if a file of any type exists with that name. And then, if a file exists with the name of the directory you tried to cd to, you decide that there are files to send (without checking to see if there are any files in that directory). Why do you log a message saying files were found when you haven't looked for any files yet???

Why not wait to log a message saying you found files to process until you actually determine that there is at least one file to process? If after searching through the directory you don't find any files to process, why not log a message saying no files were found before attempting to sftp zero files?

Every time you copy a file (locally with cp or remotely with sftp ), you should check the exit status. Instead of doing that, you assume that the copy succeeded and blindly remove your oriiginal. And at the end, why copy a file and remove the originals? Why not just move the file to the new name you want it to have? (That way, you don't risk running out of space for the copy, you don't have to worry about losing the original file contents if the copy failed. And, you'll know that you still have the contents of your original file either with the old name or the new name!)

Shouldn't the removal of the list of files to be copied and the files that you tried to copy to the remote system be kept until you have determined that the sftp completed successfully?

Aia · November 4, 2014, 11:46pm

don cragun:

Aia,
No. No. No!

The only three ways for the command:
cd $READDIR
to silently change directory to a user's home directory:

READDIR is unset or set to a blank string,

READDIR expands to an absolute pathname naming the user's home directory, or

READDIR expands to a relative pathname that moves up or down within the file hierarchy from the current working directory to the user's home

I'm sorry, but yes, yes, and yes. You just explained my point, regardless of what you thought you read on my post.

Don_Cragun · November 5, 2014, 12:14am

You said that cd /home/foobaruser/TDE will change the current working directory to the user's home directory if it is unable to change the directory to /home/foobaruser/TDE . That will NOT happen! After running the command:

cd /home/foobaruser/TDE

the current working directory will either be the directory /home/foobaruser/TDE or the current working directory will be unchanged.

Aia · November 5, 2014, 12:27am

@No Don,
I did not

This is what I said:

Noticed that I only posted a small portion of the code to just bring attention to the possible harmful practice, tenting fate with rm * . I did not even do reference to the specific code, except for the $READDIR because it was there. I could had said cd $NOTHING_HERE and it would had done the job. That portion of the code was copied to help the OP know what I was talking about.

You described the specific condition I pointed out, in your point 1:

But any of the others, would nicely had created the same disaster.