Help with UNIX "File Open" Error

We have a script that runs and picks up some files to later be used in Oracle. Every now and again we get a ERROR 'filename' file is open We know for a fact the files are not open. The files are created early morning and not used until the next morning by the script. If we manually open the file then close it, the script will pick it up with no errors.

Anyone run into this issue? I am willing to post the script if need be.

Thanks,
Randy

How does this script determine if a file is open?

1 Like

We are using:

 
if [ `fuser $i 2>&1 | awk '{ printf "%s",$2 }' | wc -c` -eq 0 ]
     then
       #FILE IS CLOSED
       echo "--------------------------------------------------------------------"

You are redirecting errors to stdout. Is there ever the possibility that fuser errors and pass the bytes alone all the way to wc, evaluating to the equivalent of "FILE IS OPEN"? YES.

Allow me to show you the concept:

% ls do_not_exist
ls: cannot access do_not_exist: No such file or directory

% ls do_not_exist | wc -c
ls: cannot access do_not_exist: No such file or directory
0

% ls do_not_exist 2>&1 | wc -c
58

% ls do_not_exist 2>&1 | awk '{printf "%s", $2}' |  wc -c
6
2 Likes

Nice catch.

1 Like

I am sorry, i shuold have posted the whole script. I am new to the forum. Thanks for all the insight and help so far. Here is the script.

sfdomain2 /u20/apps/arms/scripts (RM01) :cat process_auth.ksh
#!/bin/ksh

 
#################################################################
#
# File Name: process_auth.ksh
#
# Function: Calls load_auth.ksh for all located auth files
#
# $Revision: 1.3 $
#
#################################################################
JOB="process_auth"
USAGE="USAGE:  process_auth.ksh {YYYYMMDD} {BANK MPS|NOVA}"
#execute .profile to get environment
. $HOME/.app_profile
fail_notify()
{
   cat $OUTF | mailx -s "FAIL: $JOB " `cat $APP_BASE/secured/failDB.mail`
   cat $OUTF > $APP_BASE/trigger/$JOB.error
   exit 1
}
#PHJ - CO 3094: failDB.mail includes Madhu and Sharma Pandyaram in the failure notification list.
if [ $# != 2 ]
then
  echo $USAGE
else
BANK=$2
cdate=$1
OUTF=$APP_BASE/log/$JOB.$BANK.log.`date '+%Y%m%d%H%M'`

#begin log block
{
if ls $APP_BASE/trigger | grep error > /dev/null
then 
  echo ">>>   Error trigger file exists"
  fail_notify;
fi
echo "`date +'%m-%d-%Y %H:%M:%S'` truncating S_AUTH table" 
sqlplus -S / << EOF
truncate table app_stratus.s_auth reuse storage;
--delete from app_stratus.s_auth;
--commit;
EOF
#build list of authorization files for parameter date
cd $IRM_INCOMING/auth
########## NOVA BANK FILE PROCESSING ##############
if [ $BANK = "NOVA" ] 
then
for i in `ls *${cdate}*`
do
# call load script with date and filename to process
  if echo $i | grep usb > /dev/null
  then
   echo "BYPASSING USB AUTH FILES - USBANK FILE, NOT NOVA"
  else
    if echo $i | grep stratus > /dev/null
    then
     echo "BYPASSING STRATUS TRANSFER LOG FILES"
    else
     if [ `fuser $i 2>&1 | awk '{ printf "%s",$2 }' | wc -c` -eq 0 ]
     then
       #FILE IS CLOSED
       echo "--------------------------------------------------------------------"
       # record attributes; dup check
       echo "`date +'%m-%d-%Y %H:%M:%S'` recording file attributes $i"
       FILEATTR=`$IRM_HOME/scripts/record_file_attr_new.ksh $IRM_INCOMING/auth $i  grep RESULT | cut -f2 -d"|"`
       if [ $? -eq 0 ] 
       then
         echo "`date +'%m-%d-%Y %H:%M:%S'` CALL load_auth.ksh $1 $i"
         cp  $IRM_INCOMING/auth/$i  $IRM_INCOMING/auth/s_auth.dat
         chmod a+r $IRM_INCOMING/auth/s_auth.dat
         sqlplus -S / << EOF
exec app_stratus.pkg_load_stratus.load_auth(to_date('$1','YYYYMMDD'),'${i}');
EOF

#         $IRM_HOME/scripts/load_auth.ksh $1 $i
         echo "`date +'%m-%d-%Y %H:%M:%S'` MOVE $i to archive directory"
         mv $IRM_INCOMING/auth/$i $IRM_ARCHIVE/auth/$i
       else
         echo "`date +'%m-%d-%Y %H:%M:%S'` $i possible duplicate file, marking as duplicate; bypassing load"
         mv $i $i.duplicate
         compress $i.duplicate
         mv $i.duplicate.Z $IRM_ARCHIVE/auth
       fi   #FILEATTR=0
     else
       #log error because file is open
       banner "* ERROR *"
       echo "`date +'%m-%d-%Y %H:%M:%S'` ERROR $i file is open"
       fail_notify;
     fi  #is file open
   fi #STRATUS_TRANSFER_LOG
 fi #USBfilename
 
done
fi #BANK

########## USB/MPS BANK FILE PROCESSING ##############
#
# get all USB files regardless of date string within filename
# parameter filedate is used for settlement date
#
if [ $BANK = "MPS" ] 
then
for i in `ls usb*`
do
  
     if [ `fuser $i 2>&1 | awk '{ printf "%s",$2 }' | wc -c` -eq 0 ]
     then
       #FILE IS CLOSED
       echo "--------------------------------------------------------------------"
       # record attributes; dup check
       echo "`date +'%m-%d-%Y %H:%M:%S'` recording file attributes $i"
       FILEATTR=`$IRM_HOME/scripts/record_file_attr_new.ksh $IRM_INCOMING/auth $i | grep RESULT | awk '{ print $2 }'`
       if [ $? -eq 0 ] 
       then
         echo "`date +'%m-%d-%Y %H:%M:%S'` CALL load_auth.ksh $1 $i"
#         $IRM_HOME/scripts/load_auth.ksh $1 $i
         echo "`date +'%m-%d-%Y %H:%M:%S'` MOVE $i to archive directory"
         cp  $IRM_INCOMING/auth/$i  $IRM_INCOMING/auth/s_auth.dat
         chmod a+r $IRM_INCOMING/auth/s_auth.dat
         sqlplus -S / << EOF
exec app_stratus.pkg_load_stratus.load_auth(to_date('$1','YYYYMMDD'),'${i}');
EOF
         
         
         
         mv $IRM_INCOMING/auth/$i $IRM_ARCHIVE/auth/$i
       else
         echo "`date +'%m-%d-%Y %H:%M:%S'` $i possible duplicate file, marking as duplicate; bypassing load"
         mv $i $i.duplicate
         compress $i.duplicate
         mv $i.duplicate.Z $IRM_ARCHIVE/auth
       fi   #FILEATTR=0
     else
       #log error because file is open
       banner "* ERROR *"
       echo "`date +'%m-%d-%Y %H:%M:%S'` ERROR $i file is open"
       fail_notify;
     fi
   
done
fi #BANK = MPS

# Do a call to app_stratus.pkg_load_stratus.auth(to_date('$1','YYYYMMDD'));
echo "`date +'%m-%d-%Y %H:%M:%S'` CALL PL/SQL: PKG_LOAD_STRATUS.LOAD_AUTH($1)" 
sqlplus -S / << EOF
exec app_stratus.pkg_load_stratus.auth(to_date('$1','YYYYMMDD'));
EOF
#Check return code for error
status=$?
echo "`date +'%m-%d-%Y %H:%M:%S'` submitting background file compression at archive site" 
} >> $OUTF
if [ $status -ne 0 ]
then
  fail_notify;
else
  echo $1 > $APP_BASE/trigger/$JOB.$BANK
fi  #STATUS
if [ $BANK = "MPS" ] 
then
compress $IRM_ARCHIVE/auth/usb*.4 &
else
compress $IRM_ARCHIVE/auth/*$1* &
fi
fi #usage
exit

Posting the relevant portion of the pertaining code is always the most useful thing that your request can have, thus thank you for that.

Still, nothing posted afterwards have had the power to amend what I have previously mentioned to you.

This evaluation representing that a file is not open:

if [ `fuser $i 2>&1 | awk '{ printf "%s",$2 }' | wc -c` -eq 0 ]

has an else clause in case that the file is open:

else
       #log error because file is open
       banner "* ERROR *"
       echo "`date +'%m-%d-%Y %H:%M:%S'` ERROR $i file is open"
       fail_notify;

One issue is that under some circumstances that evaluation is misrepresented, if fuser errors, because, the error stream pollutes the stdout stream with unwanted information.

Let me put it in another words: you have a bug.
2>&1 should not be there in this case
You could use 2> /dev/null

Perhaps, it would be helpful if you can look into: why to avoid the use of `ls` in a for loop.

That's a link to information I found for you, concerning the "gotchas" of parsing the content of `ls` in a loop.

1 Like

I think aka has a good point, but there is a slight complication to this, since according to the man page:

So when stderr is redirected to stdout ( 2>&1 ) the filename with a colon and a space is prepended to the process numbers and a newline is added at the end

If stderr is redirected to /dev/null ( 2>/dev/null ) then the first PID is no longer in column 2 (also there is no longer a closing newline) , so then the awk part is no longer required (nor will it work properly), so instead one could try:

if [ $(fuser -- "$i" 2>/dev/null | wc -c) -eq 0 ]

or

if [ -z "$(fuser -- "$i" 2>/dev/null)" ]

-- is used here, so that file names that start with a - cannot break the command... ( The pattern used in the for loop for i in *${cdate}* would leave that possibility open, whereas with for i in usb* that would not be strictly necessary, but it wouldn't hurt either)