Log file is not getting created & unable to grep error from it

Shell Programming and Scripting
1

Hi All,

Below is my code,what I am trying to do is redirecting output of ftp to a log file & then greping the errors but here I am unable to grep "Permission denied" error only & also the corresponding log file is also not getting created.

#!/bin/sh

. cfg

USER='abc'
PASSWD='abc123'

MPGS_FILE_DIR_ON_LOCAL_MACHINE=$MNP_HOME

PORTIN_FILE_PREFIX='MNP_PORT_IN'
PORTOUT_FILE_PREFIX='MNP_PORT_OUT'
ACKPATTERN='.ack'
PORTIN_ACKDIR='/mpgapp/kenan/port-in/order_ack/ready'
PORTOUT_ACKDIR='/mpgapp/kenan/port-out/order_ack/ready'

COMPLETIONPATTERN='.comp'
PORTIN_COMPLETIONDIR='/mpgapp/kenan/port-in/order_comp/ready'
PORTOUT_COMPLETIONDIR='/mpgapp/kenan/port-out/order_comp/ready'

REJECTPATTERN='.rej'
ERRORPATTERN='.err'
PORTIN_ERRORDIR='/mpgapp/kenan/port-in/order_err/ready'
PORTOUT_ERRORDIR='/mpgapp/kenan/port-out/order_err/ready'

FTPLOG='/SYSTEM/custom/data/MNP/mpgsdev/log/ftp.log'
LOG='/SYSTEM/custom/data/MNP/mpgsdev/log/error.log'

INTERVAL=100

sleep 10 # wait until parent process has created log file and written pid to it
echo to exit script: kill \<pid\>

cd $MPGS_FILE_DIR_ON_LOCAL_MACHINE

# check for port-in and port-out acknowledgement, completed, error and rejected files
while true
do
date

# check for acknowledgement files for port-in
echo Checking for port-in acknowledgement files
for file in `ls $PORTIN_FILE_PREFIX*$ACKPATTERN 2>/dev/null`
do
ping $PRIMARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable' >> $LOG
ping $SECONDARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable - SUPPORT PERSONNEL TO CHECK' >> $LOG
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $SECONDARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTIN_ACKDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $PRIMARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTIN_ACKDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
done

# check for acknowledgement files for port-out
echo Checking for port-out acknowledgement files
for file in `ls $PORTOUT_FILE_PREFIX*$ACKPATTERN 2>/dev/null`
do
ping $PRIMARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable' >> $LOG
ping $SECONDARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable - SUPPORT PERSONNEL TO CHECK' >> $LOG
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $SECONDARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTOUT_ACKDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo " ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $PRIMARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTOUT_ACKDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
done

# check for completion files for port-in
echo Checking for port-in completion files
for file in `ls $PORTIN_FILE_PREFIX*$COMPLETIONPATTERN 2>/dev/null`
do
ping $PRIMARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable' >> $LOG
ping $SECONDARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable - SUPPORT PERSONNEL TO CHECK' >> $LOG
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $SECONDARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTIN_COMPLETIONDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $PRIMARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTIN_COMPLETIONDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
done

# check for completion files for port-out
echo Checking for port-out completion files
for file in `ls $PORTOUT_FILE_PREFIX*$COMPLETIONPATTERN 2>/dev/null`
do
ping $PRIMARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable' >> $LOG
ping $SECONDARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable - SUPPORT PERSONNEL TO CHECK' >> $LOG
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $SECONDARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTOUT_COMPLETIONDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $PRIMARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTOUT_COMPLETIONDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
done

# check for error and reject files for port-in
echo Checking for port-in error and reject files
for file in `ls $PORTIN_FILE_PREFIX*$ERRORPATTERN $PORTIN_FILE_PREFIX*$REJECTPATTERN 2>/dev/null`
do
ping $PRIMARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable' >> $LOG
ping $SECONDARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable - SUPPORT PERSONNEL TO CHECK' >> $LOG
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $SECONDARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTIN_ERRORDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grepb "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $PRIMARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTIN_ERRORDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
done

# check for error and reject files for port-out
echo Checking for port-out error and reject files
for file in `ls $PORTOUT_FILE_PREFIX*$ERRORPATTERN $PORTOUT_FILE_PREFIX*$REJECTPATTERN 2>/dev/null`
do
ping $PRIMARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable' >> $LOG
ping $SECONDARY -n 3
if [[ $? != 0 ]]; then
    date '+%Y-%m-%d %H:%M:%S Connection Unavailable - SUPPORT PERSONNEL TO CHECK' >> $LOG
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $SECONDARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTOUT_ERRORDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
else
    date '+%Y-%m-%d %H:%M:%S Connection Available' >> $LOG
    echo Transferring file $file
ftp -n $PRIMARY 2>&1 1> $FTPLOG <<END
quote USER $USER
quote PASS $PASSWD
cd $PORTOUT_ERRORDIR
put $file
END
EXITFTP=$?
if test $EXITFTP -ne 0; then echo "ERROR FTP" >> $LOG; exit 3; fi
if (grep "No such file or directory." $FTPLOG); then echo "ERROR FTP NO SUCH FILE" >> $LOG; fi 
if (grep "Login incorrect." $FTPLOG); then echo "ERROR FTP LOGIN" >> $LOG; fi
if (grep "Permission denied" $FTPLOG); then echo "ERROR FILE OR DIR PERMISSION ISSUE" >> $LOG; fi
fi
done

sleep $INTERVAL # sleeps for n interval

done

exit 0

Please point out where I am going wrong.

Thanks

2

move the "2>&1" to the end:

ftp -n $SECONDARY 2>&1 1> $FTPLOG

Are you sure that the log is not being created, or just that the error didn't get there?

3

You appear to be exiting the entire script if ftp returns an error code. This is before looking at $FTPLOG for the reason.
For example:

What is in /SYSTEM/custom/data/MNP/mpgsdev/log/ftp.log ?

What is in /SYSTEM/custom/data/MNP/mpgsdev/log/error.log ?

Points about creating the ftp error log:

I'd prefer this:

ftp -n $SECONDARY 2>&1 > $FTPLOG <<END

Or preferably (if you are prepared to clear the log after each instance of ftp). I have no idea why this works better with some versions of ftp but it does.

ftp -n $SECONDARY 2>>${FTPLOG} 1>>${FTPLOG}  <<END

Comments:
1) This script could be approximately one twelfth the size by putting repeated code into functions.

2) Each if the following statements would be better written as a "while" not a "for" because they will fail for large numbers of files or filenames containing space characters:

ls pattern 2>/dev/null | while read file

3) With most mainstream ftp implementations you can avoid holding passwords in a script by using a ".netrc" file. However because you are using "quote" I deduce that the target system is not unix.

4

Thanks methyl, for your comments. :slight_smile:
I incorporated them & now the log file is getting created.

Now I am not checking the ftp exit status instead just putting everything in one log file & greping the errors from it & putting in another log file.

/SYSTEM/custom/data/MNP/mpgsdev/log/ftp.log - Everything from ftp

/SYSTEM/custom/data/MNP/mpgsdev/log/error.log - Relevant errors from ftp

Need one more suggestion from you, once the files are ftp'ed i want them to move to filename.done in the same directory from where I am reading the files but this should only happen when there are no errors in ftp i.e. the file has actually ftp'ed then only it should be renamed to .done.

Could you please advice.

Thanks

5

1) You could ftp the file under a temporary name and rename the target file with a ftp "rename" at the end of the process. Then use a ftp "dir" command to show that the file exists. Then rename the source file.

2) Check that your error file exists but is empty.