Md5sum script

Shell Programming and Scripting
1

Hello,

I need to download multiple files from an FTP server but occasionally they arrive in error so I need to perform an integrity check. I've been attempting to write a bash script that does the following:

  1. Downloads all files including those in sub directories
  2. Perform md5sum using stored hash values
  3. Download any files that fail the integrity check again
  4. Repeat steps 2-3 until all files have passed the integrity check

I'm stuck on step 3. I don't know how to download the individual file so that it goes into its relevant sub-directory by itself. The relevant parts of my files and script are as follow:

checksums.txt
abcdef123456 /a/one.txt
ghijklm67890 /a/b/two.txt

file=`awk "NR==$i{print}" $checksumFile | awk -F " " '{print $2}'`

curl ftp://www.test.com/$file --user user:password

This ofcourse stores the file in the working directory. For instance, I'd like two.txt to save to /working directory/a/b/. Any ideas? Additionally, is there a better way of doing this?

Thanks

2

you can check a file like that with md5sum -c, it gives output like this:

 $ md5sum -c checksum.txt 2>/dev/null
a: OK
b: FAILED
c: FAILED open or read
d: OK

so you could handle it like this:

while ! md5sum -c checksum.txt > checklist
do
        awk -F: '!/OK$/ && $0=$1' checklist | while read FILENAME
        do
                echo "Need to download $FILENAME"
        done
done
1 Like
3

Thanks Corona. However, it doesn't explain how to download the file so that they go into their relevant directories. For instance (and assuming these files have failed on their first attempt):

one.txt needs to go in working directory/a/
two.txt needs to go in working directory/a/b/

Additionally, how do I get it to create the directories automatically?

Thanks

4 
curl --create-dirs -o a/b/c.txt

... will create the paths as it goes.

5

Works, thanks.