Need to automate sftp and always get the latest file

Shell Programming and Scripting
1

hi guys the script is working but i need to change the file that i want to get to the latest.. i will use this as cron so i can get the latest sftp file evryday.
pls help..

set user "big_user"
set IP "XX.XX.XX"
set dir "/export/home/oracle/REF/INPUT"

spawn sftp $user@$IP
expect sftp>

send "cd $dir\r"
expect sftp>

send "get /export/home/oracle/REF/INPUT/3G_20141112*.csv\r"
expect sftp>

send "exit\r"
expect eof
2

Can you generate and exchange SSH keys? It will make the whole thing much simpler. Eventually, you probably need a three step process:-

  1. SFTP list all the relevant remote files
  2. Determine the file you want to get
  3. SFTP get the file

If you exchange SSH keys then the SFTP steps become much easier.

Robin

1 Like
3

all the files are .csv's. i just need to get the latest file. and cron it to automate daily reports.

4

So can you follow the logic for each of the steps and get a working solution? If the approach is acceptable, let us know how far you are getting on each step and what you have tried. It's easier to work with your code so we can suggest adjustments in a way that you will understand and be able to support for your future. It will also show us what tools you favour so we can meet that too.

Regards,
Robin

5

the script above is what i am using,
can u help enhance it?
i dont really know what to do..

thanks so much

6

For small and few files, mget them all and delete the unwanted locally.

7

Well, it would be polite to use a correct name when it's given. I'm Robin.

If you want to get one file only rather than to get them all and delete the unwanted, the command dir within sftp will help you list them out. Capture the output of sftp to a file and work with that. If the file name structure is sufficient (you seem to have a name that includes YYYY MM DD, which is great) then that makes it easy.

As an alternate, try ls -lrt within sftp and see if the remote server will list them newest-last for you. I don't think it's always supported though. You can then trim the output to get the file name you need back in normal shell script and then pass this name into a second sftp to get the required file.

Something like:-

#!/bin/ksh
sftp user@host <<-EOSFTP >/tmp/sftp.listlog
cd /export/home/oracle/REF/INPUT
ls -lrt 3G_*
EOSFTP

tail -1 /tmp/sftp.listlog | read perm links owner group size dte1 dte2 dte3 file

sftp user@host <<-EOSFTP >/tmp/sftp.xferlog
cd /export/home/oracle/REF/INPUT
get $file
EOSFTP

Error checking would be sensible, but this will prompt, so exchange of SSH keys would be needed to automate it. Using expect might get you round it, but it is not the preferred way.

Do you know how to generate & exchange SSH keys? We'd need to know which OS the local & remote servers are running.

Regards,
Robin

8

thanks for your help Robin,,,
thanks so much