Hello All.
We are getting one zip file from another server daily. The size of the zip file will be around 4.5 gb that takes time to download completely. I have to process that file furthest once get downloaded completely. I have written one code for that, but i need suggestion whether it is a right approach or any better way with no surprises. Kindly suggest
Thanks in Advance
I am comparing size of file after every 60 seconds.
#creating function
_func() {
x=`du -sk "$path" | awk '{ print $1 }'`
sleep 5
y=2
while [ "$x" != "$y" ]
do
y=`du -sk "$path" | awk '{ print $1 }'`
echo "size before sleep is $y"
echo "sleep for another 60 seconds"
sleep 60
x=`du -sk "$path" | awk '{ print $1 }'`
echo "size after sleep $x"
done
}
#using function
if _func ; then
echo "downloading is complete"
else
echo "downloading"
fi
You can use "stat" to get the exact size of the file.("du" is quite the same here).
If you want to be sure the file is correctly downloaded, create a checksum at the source and verify the checksum after downloading. Use "md5sum" or the "sha1sum" command-family for this.
In addition to what stomp already said... Download the checksum in a separate file AFTER the download of the main file completes on the server that is downloading the files. Then just look for the 2nd file containing the checksum on the receiving client instead of monitoring the download of the 1st file.
Thanks stomp
Hello Mr. Don, the third party is sending us a zip file , presently they are not sending checksum but it is a good idea to ask them also to send checksum. But i don't understand by your statement
Download the checksum in a separate file AFTER the download of the main file
. I guess i need to compare checksum of the file at our end with the checksum of what the other party would send ? . In that case also i need to watch continuously if the checksum have matched or not by using the same while loop. I also don't understand why not to monitoring the file.
Let us propose that your 3rd party send you a zip file named file.zip (as they are doing now) and after they have sent you file.zip they then send you a file named file.checksum containing the checksum for file.zip . I am suggesting that instead of trying to figure out when file.zip is complete by trying to determine when it has stopped growing, you instead just watch for non-zero length files with names ending in .checksum . For each non-zero length file.checksum you find you will know that you have a complete corresponding file.zip and you can verify that it was received correctly by comparing the contents of file.checksum to the checksum of the file file.zip .
Thanks Mr Don, this seems very valid approach to wait for completion of transfer of main file by watching it's checksum file that will arrive at end. But just for a knowledge how third party will handle this. Presently they are manually uploading file to our end, So they can also send the checksum file for now but how this can be automated. How code can recognise if the uploading is completed and checksum file can be transferred.
Thanks.
Maybe it would be sufficient to have the 3rd party send the file as file.xxx and then on completion, they could rename it to be file.zip or whatever you agree it to be. That way you have to just test if the file exists to process it. You could just use simple filenames or have a loading directory that the file is then moved from by the 3rd party when the transfer completes.
Of course, this relies on the 3rd party process to send then rename the file.
If they are using a GUI, (e.g. Filezilla) then you could ask them to send a simple flag file after the data being transmitted. They would need to ensure that they have their queue limited to one transfer at a time, else they could get out of sequence. The flag file might need to be named something like file.zzz to ensure it follows the data file, but it is less satisfactory using a GUI because there are more risks of transferring the flag file early.