bash script - sftpbatchfile - stop on failure

Shell Programming and Scripting
21

scp should also be available, if that helps.

22

i know, but our login is only allowed through sftp, all other protocols are blocked. :slight_smile:

(if this many experts don't know the answer, i'm feeling better already about not knowing it myself :stuck_out_tongue: )

*** edit ***
my bad, it is not the protocol which was blocked. It is the user we got to log on is restricted to have only sftp acces :slight_smile:

23

Okay, possibly an outlandish punt, but could you get all the files you want to send into a single compressed tar file, delete those from local disk and when you are happy that is complete, binary SFTP the compressed tar file? Of course the other end would have to know that's what you are doing and be ready to uncompress and extract.

Just trying to think of alternatives. Sorry if it's no use. :o

Robin
Liverpool/Blackburn
UK

24

He has no ssh access, hence no way to extract it on the far end.

25

They are the same protocol, literally. They use the same port and same authentication system, and do identical things -- transfer files -- in an identical way.

It's simple enough to block shell logins, but blocking scp without blocking sftp? Strange and unlikely. If anything, they're more likely to have scp than sftp, sftp being kind of optional.

26

I know that there is a no SSH to drive it himself, however the other end is moving the files off as they arrive, so perhaps that can be altered to expect a compressed tar file, was my thought.

It depends what the receiving company is prepared to do after they "fixed" a working system that allowed Kerberos to send single files and therefore delete single files in a controlled way. Unfortunately now you want to open a single connection and move all of the files.

Another alternate would be to fire in something like into the sftp batch file:-

put localfile1 remotefile1
!rm localfile1 || kill `ps -f|grep sft[p]|cut -f3 -d " "`
put localfile2 remotefile2
!rm localfile2 || kill `ps -f|grep sft[p]|cut -f3 -d " "`
put localfile3 remotefile3
!rm localfile3 || kill `ps -f|grep sft[p]|cut -f3 -d " "`

This will cause sftp to get a kill -15 and exit with $?=1 You can then pick that up with your shell script.

Does that help?

Robin
Liverpool/Blackburn
UK

1 Like
27

i will try this one.

at the moment i had this

echo "Start of actual procedure"
echo "========================="
echo "Searching files other then ."$filetype" in "$tbtsftp" and move them to /tmp/invalid_filetype"
mkdir -p /tmp/invalid_filetype
find $tbtsftp -type f -not -name "*."$filetype -exec mv {} /tmp/invalid_filetype/ \;
echo
echo
echo "Deleting old commandlist if exists"
echo > $commandlist
echo
echo "Deleting old checkfile"
echo > $SFTPCHECK_PUT_RM
echo
echo "Creating commandlist"
echo
for FILE in `ls -tr $tbtsftp`; do
        echo "put "$FILE
        echo "cd "$remotepath_VAN >> $commandlist
        echo "put "$FILE >> $commandlist
        echo "!rm -f "$FILE
        echo "!rm -f "$FILE" | echo $? >> "$SFTPCHECK_PUT_RM  >> $commandlist
        
done

but appearantly he wont display echo $? (how it should) in my commandlist but echo 0 :s

28

What is the purpose of the pipe there? echo reads nothing from standard input.

Did you perhaps mean || ? remove the file and, on failure, print the error code?

29

The purpose would be to execute the rm and write the exit code of the rm to a file on the local machine. (sorry fairly new to scripting)

so you mean like this then?

 echo "!rm -f "$FILE" || echo $? >> "$SFTPCHECK_PUT_RM >> $commandlist
30

Ah, I see.

Your quotes had me confused, because quotes don't work that way. You weren't getting the string you expected in the file.

And the variables substituted immediately, not when the line was actually ran.

And | still doesn't make sense there.

How about:

echo '!'"rm bs2.html  -f \"$FILE\" || echo \$? >> \"$SFTPCHECK_PUT_RM\"" >> $commandlist
1 Like
31

This works great, but i cant implement it from my boss however :frowning: :wall:
he thinks it is too risky just killing a process like that every time it fails :s (read as, he thinks a local remove of file which you are owner of will fail a lot, sigh)

---------- Post updated at 09:37 AM ---------- Previous update was at 09:35 AM ----------

just tested and it seems to work :smiley:
so if i get this right, you escape the quotes and the $ sign. Why didnt I think of that :stuck_out_tongue:
will post the whole scripts when i tested completely.
fingers crossed now :slight_smile:

*** edit ***
just 1 more question which i cant seem to find the answer to.
What is exactly the difference between | and || ?

32

|| is a control operator that specifies what to do if the test is false.
|| actually represent the (logical) OR operator
[ -x file ] || echo "file is NOT executable"
if the file is NOT executable then (( [ -x file ] )) command returns a non-zero exit status code.
and if this status code is non-zero value then (( echo "file is executable" )) command is executed by shell.
for example

# touch file
# ls -ltr file
-rwxr--r-- 1 root root 52 2012-04-06 21:23 file
# chmod -x file  ## remove the executable permission
# [ -x file ] || echo "file is NOT executable"
file is NOT executable

| is a control operator that create a pipeline structure.
rpm -qa|grep glibc
in this (( grep )) reads the previous command's ((rpm -qa)) output via a pipe (|) as input for itself ((grep glibc)) and process on it.
for example

# rpm -qa
..........
........
gtkspell-devel-2.0.11-2.1
gtk+-devel-1.2.10-56.el5
beecrypt-4.1.2-10.1.1
elfutils-devel-static-0.137-3.el5
..................

/* you can think like this */

# grep glibc "..........
........
gtkspell-devel-2.0.11-2.1
gtk+-devel-1.2.10-56.el5
beecrypt-4.1.2-10.1.1
elfutils-devel-static-0.137-3.el5
.................."

grep searches the glibc in the ((rpm -qa)) output and write the results..

regards
ygemici

33

ok guys,
thanks a lot for the help. The 2nd part of my scriptseries is finished. (couldnt have done it without the help)

here is the final result :wink:

#!/bin/bash
## created by: Tommy Dekoster
## on: 15/03/2012
## Purpose: This script will be used om SFTP server via autosys to put files from acces & transit (.edi files)on the VAN server
## exit codes
## exit 6: the sftp connection to VAN server failed
echo "Definition of different exit codes"
echo "=================================="
echo "exit 6 = sftp connection to the VAN server failed"
echo

## variables
## vanuser: username used to connect to the VAN server from the SFTP server
## vanserver: Ip adress or hostname of the VAN server
## tbtsftp: to be treated folder on SFTP (files to be sent from sftp to VAN)
## remotepath_VAN: Directory on the VAN server where files need to be put
## commandlist: List of commands that will be executed over SFTP connection between SFTP1 and VAN server
## SFTPCHECK_PUT_RM: file where the last executed command from the commandlist will be
vanuser=$1
vanserver=$2
tbtsftp=$3
remotepath_VAN=$4
commandlist=$5
SFTPCHECK_PUT_RM=$6
filetype=$7
echo "Definition of different variables"
echo "================================="
echo "vanuser: username used to connect to the VAN server from the SFTP server"
echo "vanserver: Ip adress or hostname of the VAN server"
echo "tbtsftp: to be treated folder on SFTP (files to be sent from sftp to VAN)"
echo "remotepath_VAN: Directory on the VAN server where files need to be put"
echo "commandlist: List of commands that will be executed over SFTP connection between SFTP1 and VAN server"
echo "SFTPCHECK_PUT_RM: file where the last executed command from the commandlist will be"
echo "filetype: the type of files wich need to be moved/sftp'd"
echo
echo "Values of variables"
echo "==================="
echo "vanuser: "$vanuser
echo "vanserver: "$vanserver
echo "tbtsftp: "$tbtpath
echo "remotepath_VAN: "$remotepath_VAN
echo "commandlist: "$commandlist
echo "SFTPCHECK_PUT_RM: "$SFTPCHECK_PUT_RM
echo "filetype: "$filetype
echo
echo

## The actual procedure starts here
## Invalid files will be moved to a directory
## The old commandlist and sftp_put_rm checklist will be cleaned
## A commandlist containing put, rm, check on rm error will be created
## The generated commandlist will be executed over sftp
echo "Start of actual procedure"
echo "========================="
echo "Searching files other then ."$filetype" in "$tbtsftp" and move them to /tmp/invalid_filetype"
mkdir -p /tmp/invalid_filetype
find $tbtsftp -type f -not -name "*."$filetype -exec mv {} /tmp/invalid_filetype/ \;
echo
echo
echo "Deleting old commandlist if exists"
echo > $commandlist
echo
echo "Deleting old checkfile"
echo > $SFTPCHECK_PUT_RM
echo
echo "Creating commandlist"
echo
for FILE in `ls -tr $tbtsftp`
        do
                echo "put "$FILE
                echo "cd "$remotepath_VAN >> $commandlist
                echo "put "$FILE >> $commandlist
                echo "!rm -f "$FILE
                echo '!'"rm -f \"$FILE\" || \$? >> \"$SFTPCHECK_PUT_RM\"" >> $commandlist
        done
echo
echo "Executing commandlist over sftp"
echo
sftpconnect=$vanuser@$vanserver
cd $tbtsftp
sftp -b $commandlist $sftpconnect 2>&1
## Build in retry if initial connection doesnt work
        if [ $? != '0' ]; then
                echo First sftp connection failed, trying again.
                sftp -b $commandlist $sftpconnect 2>&1
                if [ $? != '0' ]; then
                        echo There is a problem connecting to the VAN server
                        echo Preforming ping
                        ping $vanserver -c 1
                        exit 6
                fi
        fi