Will shell script executes in sequence

Shell Programming and Scripting
1

I have a shell script scheduled in cron job to run at every 1 minute which transfers files to remote machine and then move the files to backup folder.

cd /u01/app/ftp_tmp
sftp user@hostname <<-EOF
cd /home/user/ftp
mput *
bye
EOF
mv /u01/app/ftp_tmp/* /u01/app/ftp_bkp

Now the problem is it transfers the file to remote machine incompletely, but it has the completed file in backup folder.
So I wanted to know how it comes to the last command before completing the file transfer. Please assist.

2

Hi,
please use code tags for your future posts.
Remove the dash before EOF.

hth

3

Several issues here:-

  • When will files be arriving? You might not have a complete file when the send starts.
  • You have no error checking, so in the event of failure you code will continue and move the file.
  • If you have hundreds of files or some large ones and the next transfer starts (you schedule every minute) what will happen with the second set of transfers when the first one completes and moves the files?

I hope that this gives you something to consider in your process.

Robin

4

Now I kept an exit code status after the file transfer , and it returns the exit code as 0
But still the file which is transferred is incomplete

cd /u01/app/ftp_tmp
sftp user@hostname <<EOF
cd /home/user/ftp
mput *
bye
EOF
if [ $? -eq 0 ]
then
echo "File transferred successfully"
else
echo "Partial File transfer : $?"
fi
mv /u01/app/ftp_tmp/* /u01/app/ftp_bkp
5

You are getting the return code of the whole SFTP. Because of the timing risks, your code may see what you consider a partial file and successfully transfer that before the file is complete - and you get a zero return code.

You would be better ensuring that a flag-file to confirm the file is complete is created and then setting up a loop to do the following for each flag-file:-

  • Delete the flag file
  • Send one file
  • Move that file to the archive

If there is a lot to transfer which may take more than a minute and your code is scheduled every minute, you may also need some sort of locking mechanism to ensure that your script is not running two versions at the same time.

Robin

6

I noticed that files larger than this are getting transferred, the issue is only with one type of file, so not sure whether the file got corrupted or not.
But each time the file gets generated with the timestamp, so the file name changes each and every time with only the third qualifer as same.

I checked with md5sum for the file both source & destination.
But since the file size in source & destination are different how to identify whether the file is corrupted or not.

7

As Robin has already said, it looks like you may have two or more copies of your cron script running simultaneously working on the same file. And, you are almost certainly starting to transfer some files before they are ready to be copied (giving you a partial copy of your file on the remote system).

You need to fix your code so that a file will only be transferred once and only after the entire file is in place and is ready to be transferred.

We don't have nearly enough information to tell you exactly how to do that since you haven't given us any details about how the files are prepared, how they are written, copied, or linked into the directory from which they are transferred; nor how you are trying to keep two or more copies of your cron job from working on the same file simultaneously.