Latest file through SFTP

Shell Programming and Scripting
1

Hi,

I want to download the latest file published on remote server using SFTP connection.
The job will run at every 15 min hence should download only the latest file. The fileName will be like ABC_DDHHMISS.txt (DD- day, HH-hour, MI - Min, SS-Sec).
So there will be files with same date for previous month, hence while downloading I should avoid old files and extract only latest files.

I tried to use the below script where the TIMESTAMP_FILE will get touched as soon as the script completes so next time when the script will run, it will fetch all new files after the timestamp of TIMESTAMP_FILE.
but when i execute the below script it's unable to find the $TIMESTAMP_FILE as it's available in local server.

Please help me to fix the issue.

NEW_FILES=`ssh username@hostname find /data/feeds/OUT -type f 
-newer $TIMESTAMP_FILE -name "ABC*"`

Thanks in advance.

2

Why not rename the files on the server or move them to a backup directory.

3

Maybe he doesn't have write access to the remote server. Otherwise he could write the time stamp file on the remote server.

4

Right, We do not have permission to write/move the files of remote server.

5

Do it in two steps. First get a listing of the files on the remote server. Then examine the list and create a list of files to retrieve.

6

Try something like this:

NEW_FILES="ssh username@hostname find /data/feeds/OUT -type f 
-newer `date -d "-15 minutes"` -name "ABC*"

with the command in back ticks.
That will check for files newer than the date - 15 minutes.

This only works with GNU date though. If they are running linux it should be gnu date

EDIT: doesn't look like you can specify a date for -newer :confused: I think the 2 step process is going to be your best bet.

7

Can it work this way -
when the script executes first time, it will fetch all the files published for respecitve date. once received to local drive, when the script executes for next interval, we can read the last file received at first run, and send the same filename as a timestamp file to check on remote server and can retrive the rest of the files published after it.

Also, when I exeute the complete script, it dosen't do anything. It just run and didnot fetch any records.

when I execute only, ssh username@hostname I am getting below message -

'Last login: Fri Mar 26 04:24:37 2010 from ny1gmcollatp1.u
Access to this computer is prohibited unless authorised
Accessing programs or data unrelated to your job is prohibited
If you are not authorised, disconnect now.'

but when i use sftp username@hostname, I am able to connect without any issues.

8

MDTM can help you here

9

Sorry, I didn't get whts MDTM means.
Can you please provide more description?

10

You can't ssh, because they are probably using an SFTP jail which only allows sftp connections not shell ssh.

I'll try to brainstorm, but you may have ot downlaod all the files and hten do your processing locally then run another get. THat being said, that would mean you would have to download them all every time which if they are large could be a problem.

Is it possibly for you to do that?

11 
#!/bin/ksh
sftp host >filelist <<EOF
cd /u
ls -lt
quit
EOF

Will put an ls -l of remote /u in your current directory.
When I tried this is I was prompted for a password after I had typed EOF.

12

Ah yes, good call on that didn't think about just outputing it to a file and then you can sort your list from there and extract only those you don't have. Now, how to do that is beyond my scripting abilities, but I'm interested in the answer.

13

while using sftp, we can only use ls -l, So the files recived to local server will not be in publish time order.

Also, where this filelist file will get generated. on local drive / remote server ?

I executed the above script but do not see any file got created with file information.

---------- Post updated at 07:56 PM ---------- Previous update was at 07:32 PM ----------

I can see the file filelist generated on local drive.
After that, I included the below code to get the latest files


###TO display the files generated in March month.

latest_file=`cat filelist | grep Mar | awk '{print $6,$7,$8,$9}'`

echo $latest_file

##To get the latest files from the above list using timestamp file.

getList=`find $latest_file -type f -newer timestamp.check 
-name "ABC*"`

echo $getList

But still I am getting the file Timestamp.check can not found which is available on my local drive.

14

Is the timestamp.check in the same directory that you are running these commands from? If not, make sure you put the full path.

Also, I don't think you can run find on the variable value of $latest_file. Y0u will probably need to find another command to extract the data.

Like getting the line, extracting the date value and comparing it to date -d "-15 minutes"

15

Create a list of files that you have received. Lets call it received_files.
Using the filelist file from the previous posts.

#!/bin/ksh
while read a b c d e f g h file
do
  grep $file received_files >/dev/null
  if [ $?  ! = 0 ]
    echo get $file  >>list_of_new_files_to_get
  fi
done<filelist
echo quit >>list_of_new_files_to_get

Maintain received_files by appending the names of the newly received files to it.
Use the list_of_new_files_to_get as input to sftp.

16

I will get the complete list everytime whenever I fetch the filenames. I am not sure whether it's possible to get only latest files from the server. That's what is the actual requirement is.

17

Yes. The filelist will have more entries in it than the current_received_list.
The script lists the names of files that are in filelist, but not the current_received_list. In other words, only the files you haven't got.

18

I think jgt's script will work, nice one jgt!

ssachins, you would run this after jgt's script: sftp -b list_newfiles username@host

you could then run your find -newer touched file command locally on your received files and append those to the received_files list per jgt's intructions

19

I will try it tomorrow and confirm.

20

Hello All,

This is what the complete script I have written.


#!/bin/ksh

sftp  user@hostname >filelist <<EOF
cd /home/data/
ls -l "ABC*"
quit

while read a b c d e f g h file
do
   grep $file Files_Receievd_ABC.txt >/dev/null
  if [ $?  != 0 ] ; then
    echo get /home/data/$file  >> list_of_new_files_to_get
  fi
done < filelist
echo quit >>list_of_new_files_to_get

list_of_new_files_to_get >> Files_Receievd_ABC.txt

sftp -v -b list_of_new_files_to_get user@hostname

### the filelist file contains first (extra sftp) record as follow due to which the script is failing.

sftp> sftp> -rw-r--r-- 0 13106 7745 0 Mar 29 05:47 ABC29054723
-rw-r--r-- 0 13106 7745 0 Mar 29 05:47 ABC29054729
-rw-r--r-- 0 13106 7745 0 Mar 29 05:47 ABC29054731
-rw-r--r-- 0 13106 7745 0 Mar 29 05:47 ABC29054732
-rw-r--r-- 0 13106 7745 0 Mar 29 05:47 ABC29054733
sftp>