We have an FTP server (vsftpd) running on Linux, that I've kinda built a "Data Management" system around. I could use some ideas as to the best way to handle/create "triggers" for file notifications.
Internal users drag 'n drop files from their Windows boxes to the server via Samba mounts. FTP users transfer files via FTP (go figure). Programs run every five minutes to check for new files. Internal files are checked by first checking the filesystem size for changes, if so, then using the find command on the FTP directory and comparing it to the previous run. On the FTP side the program checks the log file for newly transferred files.
From the difference between the two it generates an email to the proper internal & external people telling them that there are files and what the names are. Also CCing FTPadmin group, then updates a database with all the info.
Ok now that you know how the files are transferred. My questions are as follows;
1) Internal Transfers. Is there a better way to find new files than using the find and comparing the differences? We've hit the 400 user mark and at peek times running find and parsing the output can be a bit slow.
2) FTP Transfers. What's the best way to determine when an FTP transaction is complete. Keep in mind that I'll have to track multiple connections starting and ending at different times.
As to question #2, I've written a simple packet sniffer in C which does nothing more than let me know when an FTP connection has been established and from what IP, then lets me know when that connection has terminated so another program can generate notifications. The problem with this is; FTP users are using arbitrary "Keep Alives" making it virtually impossible to determine inactivity to be able to disconnect the user. To add to the complexity, the FTP server is using SSL encryption so dumping the packets to determine their contents and file progress is useless. Also, I could have vsftpd log ALL FTP events to the log file but the increase would more than triple the log file size again slowing things down to a crawl.
1) Each time you process the directory, touch a flag file. Then use find /pathname -newer /path/to/flagfile. Doesn't really save you the overhead of running find but saves you having to do comparisons.
2) Does vsftpd keep the file open for the entire duration of the transfer? If so you could use fuser on the file until it shows no processes accessing it, and then assume that the upload is complete. Normally the best way is to upload files to a temporary name and then rename them when complete, but that requires end-user actions which probably aren't appropriate in your situation.
I'm kind of surprised though that vsftpd doesn't log an upload complete event, even at a not very verbose level?
Incidentally, if you are prepared to consider a commercial product, Tumbleweed's SecureTransport is a good product for doing most of what you describe... i.e. triggered actions based on incoming files, etc.
The problem with that is files dragged from Windows don't always update the time stamp on the file(s) being transferred. I've tried using the atime, mtime, and ctime of the files, none of which would work 100% of the time.
Now that's a good question. I don't know, didn't even think of it, but I can check on it. Thanks!
vsftpd does log when the transfer is complete. But we have users that will transfer 5 - 15 files at times. I don't really want a notification for each file, just one when the lot is complete. So for now when it sees a complete transfer with sleeps for 15 seconds, checks the filesystem size for changes, if yes waits for the next transfer to complete, if not, send the email.
I've not heard of Tumbleweed, nor been able to find much of anything like what I'm doing. Thanks! I'll check it out, but I know the company won't spring for it until the economy picks up. I am in an auto related field.