Problems with expect and sftp in bash

Shell Programming and Scripting
1

I'm having trouble with some automated sftp pulls. I'm using expect inside bash scripts and spawning SFTP. Some times the expect seems bog down. I have tried to put sleeps in my code to give everything time to work before I move on to next step but I till continue to get issues. For example when the function removeFlagFiles is called a for loop is used to get the name of each flag file downloaded. The expect part of the loop is supposed to delete the flag file. I can watch the output and all of a sudden the interaction seems to slow way down and sometimes the file is not deleted. Can someone suggest better ways of doing this?

#!/bin/bash
#
# Author: Gene Osteen
# Date: 12/11/2014
# Description: Pull performant HUMA files directory(s) and flag file(s).
 
# Hard-coded directories that have to be in the current (running) directory.
FILTER="HUMA*.flag"
DOWNLOAD_DIR="/ftpdata/HUMA/incoming/HUMA.dir"
REMOTE_DIR=""

function pullDataFile {
echo "In pullDataFile" >> $4
echo "remote dir = $1"
echo "file name = $2"
echo "dir = $3"
echo "remote dir = $1 file name = $2 dir = $3\n"  >> $4
maxiter=5
iter=1
filestate="trypull"
while [ $filestate = "trypull" ]
do
echo "================= Start SFTP pull of $2 iteration $iter ===================" >> $4
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
sleep 1
expect "(yes/no)?"
sleep 1
send "yes \r"
sleep 1
expect "password:"
sleep 1
send "somepassword\n"
sleep 1
expect "sftp>"
log_file $4
send "cd $1 \r"
expect "sftp>"
send "lcd $3 \r"
expect "sftp>"
send "get $2 \r"
expect {
    #Check for progress, note does not work with all versions of SFTP
    #If a match is found restart expect loop
    -re "\[0-9]*%" {
        set percent $expect_out(0,string)
        #puts $logf "File transfer at $percent, continuing..."
        exp_continue
    }
    #Check for common errors, by no means all of them  
    -re "Couldn't|(.*)disconnect|(.*)stalled" {
         #puts $logf "Unable to transfer file"
         exit 1
    }
    #OK continue
    "sftp>" {
         #puts $logf "File transfer completed"
    }
}
log_file
send "quit \r"
EOF
sleep 3
if [ -f ./$3/$2 ]; then
 echo "File ./$3/$2 Arrived" >> $4
 filestate="Arrived"
else
    echo "File ./$3/$2 HAS NOT ARRIVED" >> $4
 iter=$[$iter+1]
 if [ $iter -gt $maxiter ]; then
  filestate="Too Many Tries"
 else
  sleep 10
 fi
fi
 
done
if [ $filestate = "Arrived" ]; then
 echo "chmod ./$3/$2 to 755" >> $4
 chmod 775 ./$3/$2
# Below must be formatted with no leading white space
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
sleep 1
expect "(yes/no)?"
send "yes \r"
sleep 1
expect "password:"
send "somepassword\n"
sleep 1
expect "sftp>"
log_file $4
send "cd $1 \r"
expect "sftp>"
send "rm $2 \r"
expect "sftp>"
log_file
send "quit \r"
EOF
 
 
else
 echo "Send Email" >> $4
 #Send email when you can
fi

 sleep 4
 
}

function removeFlagFiles {
echo "======================= Remove Flag Files =======================================\n\n" >> $1
for f in $FILTER;
do
 echo "Remove $f" >> $1
 delfile="$f"
 delfile2="${delfile//\#/\\\#}"
 echo "delfile2 = $delfile2"
 echo "delfile = $f delfile2 = $v" 
 sleep 3
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
sleep 1
expect "(yes/no)?"
sleep 1
send "yes \r"
sleep 1
expect "password:"
sleep 1
send "somepassword\n"
expect "sftp>"
sleep 1
log_file $1
send "cd outgoing \r"
sleep 1
expect "sftp>"
send "rm $delfile2 \r"
sleep 1
expect "sftp>"
log_file
send "quit \r"
EOF
 sleep 4
done
echo "======================= Finished Removing Flag Files =======================================\n\n\n" >> $1
}
function pullFlagFiles {
echo "======================== Pull existing flag files===========================================" >> $1
echo "Start of SFTP" >> $1
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
expect "(yes/no)?"
send "yes \r"
expect "password:"
send "somepassword\n"
expect "sftp>"
log_file $1
send "cd outgoing \r"
expect "sftp>"
send "get $FILTER \r"
expect "sftp>"
log_file
send "quit \r"
EOF
echo "======================== Finished Pulling existing flag files===========================================\n\n\n" >> $1
sleep 4
}

function processFiles {
for f in $FILTER;
do
 echo "======================== Process File $f =================================\n" >> $1
 arr=($(echo $f | tr "#" "\n"))
 dir=${arr[@]:1:1}
 fname=${arr[@]:2:2}
 fname2="${fname/.flag/}"
 remote_dir="outgoing/HUMA.dir/$dir"
 echo $dir
 echo $fname2
 echo $remote_dir
 newdir="$DOWNLOAD_DIR/$dir"
 echo "newdir = $newdir"
 echo "newdir = $newdir" >> $1
 
#   Create the directory if it's not there already.
 echo "Try to create $newdir"
  echo "Try to create $newdir" >> $1
 
 mkdir -p $newdir
 status=$?
 echo "status from mkdir $newdir = $status" >> $1
 if [ -d $newdir ];
 then
    echo "$newdir exists."
    echo "$newdir exists." >> $1
 else
    echo "File $newdir does not exist."
    echo "File $newdir does not exist." >> $1
    return 
 fi 
 
 chmod 775 $newdir
 echo "Target Directory = $newdir\n" >> $1
 
 # Get data file 
 pullDataFile $remote_dir $fname2 $dir $1
 
 if [ -f $dir/$1 ];
 then
    echo "File $dir/$1 exists." >> $1
 else
    echo "File $dir/$1 does not exist." >> $1
 fi 
 
#   Change the permission and Move the file into the directory where it's wanted.
 chmod 775 $f
 
 echo "mv $f $newdir" >> $1
 mv $f $newdir
 
 if [ -f $fname2 ];
 then
    echo "File $newdir/$fname2 exists." >> $1
 else
    echo "File $newdir/$$fname2 does not exist." >> $1
 fi 
 
 
 
 
 echo "==================== Finished File $f ===========================================\n\n\n" >> $1
done
}

###################
####### MAIN ######
###################
cd /ftpdata/HUMA/incoming/HUMA.dir
DATE=`date +%Y%m%d`
echo "DATE = $DATE"
logfile="./Logs/ftp_log_$DATE.log"
echo "Log_file = $logfile"
if [ -f ./HUMA_running.txt ]
then
    echo "Already running"
 exit
fi
echo "Let's start"
touch HUMA_running.txt

echo "Start HUMA search : $(date)"  >> $logfile
# Now do a full cycle of pull, rename, and process.
echo "pullFlagFiles"  >> $logfile
pullFlagFiles $logfile
 
ls -l $FILTER > /dev/null 2>&1
if [ "$?" = "0" ]; then
        removeFlagFiles $logfile
        processFiles $logfile
else
        echo "No files to process" >> $logfile
fi
echo "" >> $logfile
rm ./HUMA_running.txt
2

Is there a reason not to exchange ssh-keys which will automate the sftp login and therefore you can exploit the functionality of sftp?

It would leave you with far neater code something like:

printf "cd /outgoing\nget $file\n" > /tmp/sftp.batch
sftp -b /tmp/sftp.batch 10.10.10.10 >$1 2>&1

It might be possible to set it up as a here document too:-

sftp 10.10.10.10 <<-EOSFTP >$1 2>&1
cd /outgoing
get $file
EOSFTP

Would this help you make things neater, easier and remove the reliance on expect?

Robin

1 Like
3

I will speak with my Admins to see if I can get ssh-keys set up.