Executing a job with file success

UNIX for Advanced & Expert Users
1

Hello All,

I have a scenario where I need suggestion

I am creating a file watcher job on autosys and and command job to do a SCP a file from one server to another. My file watcher will get success once the file get receive. The file I am receiving a zip file. THe command job which I am running will be running every 10 min to receive the file from the server. Now I am not sure what will happen the SCP job triggered when the zip file is halfdone. Is there a way that I can do a check for the zip to make sure it is completed .

Thanks
Arun

2

The only reliable way to do this is to have the uploader inform you when it is done. Often this is done by using sftp(same network protocol as scp) to first put the file into an 'uploads' folder, then mv it to the real destination somewhere in the same partition. This guarantees that only 100% complete files will appear for your file watcher.

1 Like
3

Have the scp process send two files, the first the actual data file, and the second a small (20 bytes) file. Whenever the second file exists, you will know that the large data file has been received completely.

4

I take it you are worried about finding the file while it is in the process of being transferred?

If you are using scp with a key you could consider using a trigger script which would be able to process the file once the transfer is complete.

authorized_keys:

command="/path/to/script" key

In the script:

if [[ ${SSH_TTY:-unset} == "unset" ]]
then
   cmd=${SSH_ORIGINAL_COMMAND%% *}
   case ${cmd} in
      scp) declare -a cmd_args=(${SSH_ORIGINAL_COMMAND#* })
           pre_scp ${cmd_args[*]}       # find existing files
           scp ${cmd_args[*]}           # perform scp
           post_scp ${cmd_args[*]}      # find new files
           ;;
   esac
fi

So the SSH_TTY variable is set for an interactive shell; you have effectively disabled using this key for interactive sessions - a bonus if this is used with a passphrase-free key pair. The pre_scp and post_scp commands are shell functions defined elsewhere in the script to identify the file(s) being copied. The post_scp function could then run the alert.

Be aware that using this will hold the scp session on the client side open; you may wish to fork a process in the post_scp function.

Andrew

5

You can test the zip archive, exit code ($?) should be zero on good file.

unzip -qtl archive.zip && printf "%s\n" "File is fine do the job.."