# sftp mget where file doesn't exist BASH

**URL:** <https://community.unix.com/t/sftp-mget-where-file-doesnt-exist-bash/232498>\
**Category:** Shell Programming and Scripting\
**Created:** [February 20, 2009, 7:54pm UTC](https://community.unix.com/t/sftp-mget-where-file-doesnt-exist-bash/232498 "2009-02-20T19:54:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![unclecameron](https://community.unix.com/letter_avatar/unclecameron/32/6_449c533f586939b6e5f246632a46421f.png) [@unclecameron](https://community.unix.com/u/unclecameron)\
**Post date:** [February 20, 2009, 7:54pm UTC](https://community.unix.com/t/sftp-mget-where-file-doesnt-exist-bash/232498/1 "2009-02-20T19:54:31Z")

</div>

I have a script that is working:

```nohighlight
#!/bin/bash
sftp user@domain.com <<EOF
cd somedir
mget *.csv
quit
EOF

```

but on a crontab I want to only pull newer files, so I want to do something like:

while read ls current dir local file != true do  
mget that new file

but I'm not sure the syntax to get sftp to check a local variable to see if a a remote file = the local one. I would really like to use rsync -auv -e ssh, but they won't give me a system account to do it, only sftp. Is there some way I can do this?

---

<div class="post-metadata">

**Author:** ![otheus](https://community.unix.com/user_avatar/community.unix.com/otheus/32/505_2.png) [@otheus](https://community.unix.com/u/otheus)\
**Post date:** [March 5, 2009, 11:26am UTC](https://community.unix.com/t/sftp-mget-where-file-doesnt-exist-bash/232498/2 "2009-03-05T11:26:38Z")

</div>

So you can run sftp but not ssh? That's unusual.

Check out [Net::SFTP - Secure File Transfer Protocol client - search.cpan.org](http://search.cpan.org/~DBROBINS/Net-SFTP/lib/Net/SFTP.pm)

Can you write Perl code? If not, search these forums for a question like this, but with FTP. Then use the SFTP module instead. The code should be nearly identical.

---

<div class="post-metadata">

**Author:** ![unclecameron](https://community.unix.com/letter_avatar/unclecameron/32/6_449c533f586939b6e5f246632a46421f.png) [@unclecameron](https://community.unix.com/u/unclecameron)\
**Post date:** [March 5, 2009, 12:30pm UTC](https://community.unix.com/t/sftp-mget-where-file-doesnt-exist-bash/232498/3 "2009-03-05T12:30:37Z")

</div>

well I wound up writing one, I know it's probably a terrific hack, but here it is in case it helps someone else, this took me awhile to figure out

```plaintext
#!/bin/bash

# pipe ls of remote folder to file
# to compare to figure out which files
# are newer
b=`sftp user@ftp.domain.com <<EOF
cd folder_with_hourly_dataset
ls
quit
EOF`
echo $b > remotelist.txt

# sanitize that file, so that all we
# see are the .csv entries
list=$(cat remotelist.txt)
for i in $list
do
    echo $i | grep \.csv$ >> sortedremotelist.txt
done

# create local list of files
ls *.csv >> locallist.txt

# compare local/remote list and then
# sanitize the output file into a file
# with one .csv per line so we can 
# use a for loop to sftp get it
diff locallist.txt sortedremotelist.txt > get
cut -c3- get > get2
grep \.csv get2 > get3

# now go get that list of files
c=$(cat get3)
for i in $c
do
    d=`user@domain.com <<EOF
    cd directory_with_hourly_data
    mget $i
    quit
    EOF`
    echo "-------------"
    echo "getting file: "$i
done

# clean up files for next time
cat /dev/null > remotelist.txt
cat /dev/null > sortedremotelist.txt
cat /dev/null > get
cat /dev/null > get2
cat /dev/null > get3

```

hope that helps someone else. Of course I could've done the whole thing with a single rsync -auv -e ssh [user@example.com](mailto:user@example.com):/folder\_with\_updates/ ./ but I couldn't get a system user account on the box to do it ☹