Hotfolder with inotify-tools, loop FOR not working

pasaico · April 5, 2018, 8:51am

I can not understand why this little script with a loop processes only one file.

At boot in /etc/rc.local i wrote:

/usr/local/bin/./myscript &

This is myscript:

#!/bin/bash
    while inotifywait -e create /HOTFOLDER/ ; do

    for fullname in /HOTFOLDER/*.xlsx; do
 
    if ! [ -d "$fullname" ]; then
            send=/HOTFOLDER/$(date +'SEND-%F')
            sleep 10s
                    mkdir -p $send
                    mv -- "$fullname" $send
    fi

    done
    done

I have try also the option -m in inotifywait, but same problem:

 while inotifywait -m -e create /HOTFOLDER/ ; do

apmcd47 · April 6, 2018, 4:22am

I know nothing about inotifywait but it looks to me like you are using it wrong.
Try something like this:

inotifywait -m -e create /HOTFOLDER/ 2>/dev/null | while read folder event file
do
   if [[ "$event" =~ CREATE ]]
   then
      fullpath="$folder/$file"
     # process fullpath here
   fi
done

In bash everything inside that loop will be in a subshell and none of your variables will survive the loop.

Andrew

Edit: Just realised the redirect is not necessary:

inotifywait -m -q -e create /HOTFOLDER/ | while read folder event file
do
   if [[ "$event" =~ CREATE ]]
   then
      fullpath="$folder/$file"
     # process fullpath here
   fi
done

pasaico · April 6, 2018, 5:47am

apmcd47:

I know nothing about
[/CODE]In bash everything inside that loop will be in a subshell and none of your variables will survive the loop.

Andrew

Edit: Just realised the redirect is not necessary:
inotifywait -m -q -e create /HOTFOLDER/ | while read folder event file
do
   if [[ "$event" =~ CREATE ]]
   then
   fullpath="$folder/$file"
   # process fullpath here
   fi
done

Thnk you for your answer. after I will try your suggestion
the problem with myscript:
If in the HOTFOLDER there are 10 .xlsx files and then I copy a new one, the loop works all file to the end.
If there is no file inside the HOTFOLDER and then I copy 10.xlsx, the loop processes only one

RudiC · April 6, 2018, 6:11am

Not seeing any reason in the script for the behaviour you describe, I think we need more context info. Please post the output of the script run with the -x option set, and also a directory listing from within every loop.

pasaico · April 6, 2018, 6:50am

Thank you Rudic for the answer

I have copied in the HOTFOLDER 1.xlsx 2.xlsx 3.xlsx 4.xlsx
as you can see, only the 4.xlsx number was processed and the others remained in the HOTFOLDER

after that I copied the 4.xlsx number back into the HOTFOLDER and now it has processed all the xlsx files present.

I highlight you in bold inotifywait waiting to receive input

this is the output of $:bash -x myscript.sh

+ inotifywait  -e create /HOTFOLDER/
Setting up watches.
Watches established. (start of myscript)
+ for fullname in '/HOTFOLDER/*.xlsx'
+ '[' -d /HOTFOLDER/4.xlsx ']'
++ date +SEND-%F
+ send=/HOTFOLDER/SEND-2018-04-06
+ sleep 10s
+ mkdir -p /HOTFOLDER/SEND-2018-04-06
+ mv -- /HOTFOLDER/4.xlsx /HOTFOLDER/SEND-2018-04-06
+ inotifywait  -e create /HOTFOLDER/
Setting up watches.
Watches established. (after copy 1-2-3-4.xlsx, as you can see it only processes number 4)
+ for fullname in '/HOTFOLDER/*.xlsx'
+ '[' -d /HOTFOLDER/1.xlsx ']'
++ date +SEND-%F
+ send=/HOTFOLDER/SEND-2018-04-06
+ sleep 10s
+ mkdir -p /HOTFOLDER/SEND-2018-04-06
+ mv -- /HOTFOLDER/1.xlsx /HOTFOLDER/SEND-2018-04-06
+ for fullname in '/HOTFOLDER/*.xlsx'
+ '[' -d /HOTFOLDER/2.xlsx ']'
++ date +SEND-%F
+ send=/HOTFOLDER/SEND-2018-04-06
+ sleep 10s
+ mkdir -p /HOTFOLDER/SEND-2018-04-06
+ mv -- /HOTFOLDER/2.xlsx /HOTFOLDER/SEND-2018-04-06
+ for fullname in '/HOTFOLDER/*.xlsx'
+ '[' -d /HOTFOLDER/3.xlsx ']'
++ date +SEND-%F
+ send=/HOTFOLDER/SEND-2018-04-06
+ sleep 10s
+ mkdir -p /HOTFOLDER/SEND-2018-04-06
+ mv -- /HOTFOLDER/3.xlsx /HOTFOLDER/SEND-2018-04-06
+ for fullname in '/HOTFOLDER/*.xlsx'
+ '[' -d /HOTFOLDER/4.xlsx ']'
++ date +SEND-%F
+ send=/HOTFOLDER/SEND-2018-04-06
+ sleep 10s
+ mkdir -p /HOTFOLDER/SEND-2018-04-06
+ mv -- /HOTFOLDER/4.xlsx /HOTFOLDER/SEND-2018-04-06
+ inotifywait  -e create /HOTFOLDER/
Setting up watches.
Watches established. (after copy 4.xlsx again, as you can see it  processes all xlsx present in HOTFOLDER)

RudiC · April 6, 2018, 7:07am

Where is the directory listing as requested? Did you copy all the files in one go, or one by one? By the way, why don't you mv all .xlsx files in one go?

apmcd47 · April 6, 2018, 8:21am

It is because of the way you were using inotifywait .

inotifywait -e create /some/dir

will wait for the create event in /some/dir, report that event and exit.

You copy 10 files into that directory and only the first is processed because the script is finished (or at least the wildcard *.xlsx is expanded) before the second file is added.

inotifywait -m -e create /some/dir | while read dir event file

will run the loop for every file added to the directory /some/dir and will process the files one-at-a-time.

Andrew

pasaico · April 6, 2018, 8:21am

I do not understand what you mean by directory list.
I copy all the file in one go
the command mv all .xlsx files in one go or not,
move files at once or not? I need it to be one by one.

RudiC · April 6, 2018, 8:33am

Yes, but shouldn't the for fullname in /HOTFOLDER/*.xlsx then run across ALL files present? Or do you think it's a race / timing problem? Why, then, would the 4th (i.e. the last) file be treated only?

ls /HOTFOLDER/

Why?

pasaico · April 6, 2018, 8:47am

apmcd47:

It is because of the way you were using inotifywait .
inotifywait -e create /some/dir
will wait for the create event in /some/dir, report that event and exit.

You copy 10 files into that directory and only the first is processed because the script is finished (or at least the wildcard *.xlsx is expanded) before the second file is added.
inotifywait -m -e create /some/dir | while read dir event file
will run the loop for every file added to the directory /some/dir and will process the files one-at-a-time.

Andrew

Andrew,
I had already tried with the -m option but it does not work, not even a file is moved

while inotifywait -m -e create /HOTFOLDER/ ; do

---------- Post updated at 02:47 PM ---------- Previous update was at 02:33 PM ----------

ls /HOTFOLDER/

HOTFOLDER is empty folder

Because this is the part of code that does not work ... there's more but it's not relevant to the problem

RudiC · April 6, 2018, 8:51am

If it is an empty folder, why should ANY file be copied from there?

pasaico · April 6, 2018, 9:00am

HOTFOLDER is a shared directory with samba, empty.

The files are copied into HOTFOLDER which is then processed using a script that automatically sends emails with attachments to the .xlsx file.

The problem is that the LOOP FOR stops responding when inotifywatch recognizes the event of a new file inside HOTFOLDER

apmcd47 · April 6, 2018, 9:13am

Somehow by the way pasaico is copying the files the fourth file is the first to be created. As soon as it is created inotifywait exits and there is a full ten seconds before it is run again. I imagine the wildcard expansion is faster than the copy which is why only one file is seen. When I tested the code 1.xlsx was copied.

Original script (modified for testing by me):

#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER

    while inotifywait -e create ${HOTFOLDER} ; do

    for fullname in ${HOTFOLDER}/*.xlsx; do
 
    if ! [ -d "$fullname" ]; then
            send=${HOTFOLDER}/$(date +'SEND-%F')
            sleep 10s
                    mkdir -p $send
                    mv -- "$fullname" $send
    fi

    done
    done

If we put that sleep statement between the while and for statements you give the copy time to catch up. Try it.

This is the solution I proposed (now properly coded):

#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER2

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do
   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
      fullpath="$folder/$file"
      send=${HOTFOLDER}/$(date +'SEND-%F')
      mkdir -p "${send}" && mv "${fullpath}" "${send}"
   fi
done

Now, every time a file is copied or otherwise created the while loop is run. I am checking for a CREATE rather than a CREATE,ISDIR and the file has the correct extension.

Andrew

pasaico · April 6, 2018, 10:18am

apmcd47:

Somehow by the way pasaico is copying the files the fourth file is the first to be created. As soon as it is created inotifywait exits and there is a full ten seconds before it is run again. I imagine the wildcard expansion is faster than the copy which is why only one file is seen. When I tested the code 1.xlsx was copied.

Original script (modified for testing by me):
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER

   while inotifywait -e create ${HOTFOLDER} ; do

   for fullname in ${HOTFOLDER}/*.xlsx; do
 
   if ! [ -d "$fullname" ]; then
   send=${HOTFOLDER}/$(date +'SEND-%F')
   sleep 10s
   mkdir -p $send
   mv -- "$fullname" $send
   fi

   done
   done
If we put that sleep statement between the while and for statements you give the copy time to catch up. Try it.

This is the solution I proposed (now properly coded):
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER2

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do
   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
   fullpath="$folder/$file"
   send=${HOTFOLDER}/$(date +'SEND-%F')
   mkdir -p "${send}" && mv "${fullpath}" "${send}"
   fi
done
Now, every time a file is copied or otherwise created the while loop is run. I am checking for a CREATE rather than a CREATE,ISDIR and the file has the correct extension.

Andrew

Maybe you're right, but the strange thing is that this script worked for months.

Now anyway
I also modified your script by inserting sleep 10s before MKDIR and it works ok

#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER2

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do
   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
      fullpath="$folder/$file"
      send=${HOTFOLDER}/$(date +'SEND-%F')
      sleep 10s
      mkdir -p "${send}" && mv "${fullpath}" "${send}"
   fi
done

thanks for the suggestion and the help

pasaico · April 9, 2018, 9:50am

apmcd47:

This is the solution I proposed (now properly coded):
#!/bin/bash
HOTFOLDER=$PWD/HOTFOLDER2

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do
   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
   fullpath="$folder/$file"
   send=${HOTFOLDER}/$(date +'SEND-%F')
   mkdir -p "${send}" && mv "${fullpath}" "${send}"
   fi
done
Now, every time a file is copied or otherwise created the while loop is run. I am checking for a CREATE rather than a CREATE,ISDIR and the file has the correct extension.

Andrew

thank you very much your suggestions have been very useful,

I would like to add the function to check the email from a text file.
If the variable $mail is present, togheter other event (CREATE and .xlsx) proceed with Echo "OK"
otherwise with echo "ERROR"

This way it does not work, I do not understand much about expression and operators

can you help me another time?

#!/bin/bash 
HOTFOLDER=$PWD/HOTFOLDER

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do

filename="${file##*/}"
name=$(echo "${filename%.*}")
mail=$(awk -F ',' -v var="$name" '$1==var {print $2,$3}' /address.txt)

if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]] && [[ "$mail" = '' ]]
        then
                echo "ERROR"

         else
                echo "OK"
         fi
  done

apmcd47 · April 9, 2018, 11:11am

What does the file at /address.txt contain?

#!/bin/bash 
HOTFOLDER=$PWD/HOTFOLDER

inotifywait -q -m -e create ${HOTFOLDER} | while read folder event file
do

filename="${file##*/}" #(1)
name=$(echo "${filename%.*}") #(2)
mail=$(awk -F ',' -v var="$name" '$1==var {print $2,$3}' /address.txt)

if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]] && [[ "$mail" = '' ]]
        then
                echo "ERROR"

         else
                echo "OK"
         fi
  done

(1) There will be no filepath component, so you do not have to strip it off.
(2) You don't need to put this in an echo:

name="${filename%.*}"

will work just as well.

Andrew

pasaico · April 10, 2018, 4:45am

address.txt contain:

1,mail@mail.com ,mail2@mail.com
2,mail2@mail.com
3,mail3@mail.com ,mail4@mail.com

if i put a file 1.xlsx it's check with $mail if present in address.txt and generate "ECHO OK"
if i put 4.xlsx it should generate "ECHO ERROR"

thank you for other suggest

apmcd47 · April 10, 2018, 6:31am

Presumably, going back to your original script, you want to move the file and email confirmation if the email address(es) exist. What happens if there is no email address? Move anyway? Or is that an error?

   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]]
   then
      echo "do the copy"
      mail=$(awk -F ',' -v var="${file%.*}" '$1==var {print $2,$3}' ${addr})
      if [[ -n "$mail"  ]]
      then
         echo "Can send email"
      else
         echo "ERROR cannot send email"
      fi
   else
      echo "do nothing"
   fi

   if [[ "$event" =~ ^CREATE$ ]] && [[ "${file}" =~ \.xlsx$ ]] && [[ -n "$mail"  ]]
   then
      echo "OK"
   else
      echo "ERROR"
   fi

In my opinion, rather than using

[[ "$var" = '' ]]

it is better to use

[[ -z "$var" ]]

for string is zero-length or

[[ -n "$var" ]]

for string is non-zero-length.

Andrew

pasaico · April 10, 2018, 9:37am

If there are no email addresses, the file is moved into a directory named ERROR

error=${HOTFOLDER}/$(date +'ERROR-%F')
                mkdir -p "${error}" && mv "${fullpath}" "${error}"

why with:

[[ -n "$mail"  ]] #work
[[ "$mail" = '' ]] #not work?

Thank you Andrew

pasaico

apmcd47 · April 10, 2018, 9:54am

I don't know. It should have.

Andrew