using ftp as a client would make things easier in that way as you don't have to mount anything. On the other hand, you have to open always one connection everytime you want to check for changes and another in the case that there is something to download. I think the curlftpfs version is better, since you can handle the dir as it were local. You have to check for changes in both cases.
The python script assumed that there were subdirs plus it handled exceptions in case of network errors, so it got a little bit longer. Here is a shorter bash version without that:
cd /path
f0=$(ls R_*)
while :; do
sleep 30 # should be sufficient
f1=$(ls R_*)
for f in $f1; do
[[ $f0 == *$f* ]] || echo "created: $f"
done
for f in $f0; do
[[ $f1 == *$f* ]] || echo "moved: $f"
done
f0=$f1
done
You can have an exact match by adding extra boundaries on both sides.
Here extra newlines, because the ls output is separated by newlines
nl="
"
...
[[ $nl$f0$nl == *$nl$f$nl* ]] || echo "created: $f"
...
[[ $nl$f1$nl == *$nl$f$nl* ]] || echo "moved: $f"
@bendingrodriguez and @MadeInGermany that works like a charm. Greatly appreciated.
I am not totally clear on the echo, since nothing is sent to the screen, but I have used logger instead, and they show up in syslog.
how / where do you call the script? Seems that stdout goes to nirvana. You could place a >&2 after the echos.
I call the script on the command line as user, ./moveit.sh...
I am planning to run this script as a service, so I don't need the echo to screen...
then I don't understand either why it doesn't log to the console 
How can I prevent that a file is being copied when it is being fed into the mounted ftp directory ?
when a file is stored on the server side, it appears in the mounted dir and vice versa. But that's not a copy, it's the file itself and physically it's always stored on the remote side.
If you umount the dir and puts a file in it, it will be stored locally. When you then mount the dir again, this file is no longer shown by ls, but is still there, only 'covered' by the server side files.
What exactly do you want to do?
Yes, I understand that. I need to download all files staring with R_ to a local directory. Your suggestion to monitor the dir and catch the files above was useful. But the concerning files can be really big (100GB) and those are fed into directory on the ftp server. I have tried cp and rsync, but the issue is that my script starts copying as the remote fle is created and not completely written in the remote dir by users.
nl=""
f0=$(ls R_*)
while :; do
sleep 10
f1=$(ls R_*)
for f in $f1; do
[[ $nl$f0$nl == *$nl$f$nl* ]] || logger "created: $f"
done
for f in $f0; do
[[ $nl$f1$nl == *$nl$f$nl* ]] || logger "moved: $f"
done
f0=$f1
echo "Copying $f0"
cp $f0 $LOCALPATH/$f0 `
done```
The file received is a different filesize as the file on the remote dir.
well, 100GB is quite a chunk... You need another loop which checks for write completion on server side, at least I don't know of any other simple method.
I've removed the newlines, since they only have to be taken into account when $f0``/$f1 are enclosed by quotes.
Note that this does not work if the filenames contain spaces. In addition, the script is not resistant to network or other failures, since it doesn't provide any error handling.
f0=$(ls R_* 2>/dev/null)
while sleep 10; do
f1=$(ls R_* 2>/dev/null)
for f in $f1; do
if [[ $f0 != *$f* ]]; then
logger "$f opened for writing, waiting for finish ..."
s0=$(stat -c %s $f) s1=-1
# if the file size doesn't change within 10s, $f will be sync'd
while sleep 10; do
s1=$(stat -c %s $f)
((s1 > s0)) && s0=$s1 || break
done
logger "syncing $f to $LOCALPATH ..."
# use rsync, since it skips already sync'd files and resumes broken syncs.
# this could take a while, since we're syncing over network.
rsync -a $f $LOCALPATH
fi
done
for f in $f0; do
[[ $f1 == *$f* ]] || echo "$f moved"
done
f0=$f1
done
Thanks to all contributing !