I receive files by FTP in an input directory. Every 10 seconds a process checks if some new file has arrived to the input directory and then the process moves them to another directory to be read by a second process.
The problem is that there is no way to know if the download has finished or not. Sometimes my process moves the file when I have received only a part of it.
Is there any way to know if the FTP download is completed ? I have tested the file with 'lsof', 'stat', ... but it doesn't work.
The same process works fine in AIX. If I try to open a file that is downloading at the same time, I receive an error 'Resource is busy'. I use this error to check if the download is completed. But in Linux ( SuSe ) I don't receive any error.
while I believe there's no "bullet-proof" solution to this.... you can try using 'fuser' [or lsof if you have one] to see if a file is being used by someone else.
true ... it might be better to combine the netstat-ftp check with fuser so there is some sort of confirmation that the file is indeed being written to ...
Also, it might be possible to ask the ftp transmitter to send another file when the first ftp is complete. It would be a 0 byte or small file and then you would just check for the existense of that file.
that might sound like the easiest option but that would mean 2 scripts have to be modified ... considering that the same script works well in another platform --- i would opt to just modify 1 script myself as that would keep the appearance of new "features" to a minimum ...
I have tested all this solutions but by now, I'm not able to get if the file is complete downloaded.
The command 'lsof' doesn't show the file that the 'vsftpd' process is writting. I get the same problem with 'fuser'.
I have made a C program that lock in exclusive mode the file. I thought that If I am able to lock the file then the file is not open by another process. But It doesn't work, the program is able to lock the file although the file is downloading.
The real problem appear to be that, nowhere in the Linux system reside information about the files than the vsftpd daemon is working with ...
My system receive files from a lot of external different systems, so I can't request to all this systems ( some of them, another companies ) a second file to confirm that the first one is completed.
Re paragraph 1: I find it very hard to believe that lsof can't do this. Your lsof could be broken, more more likely you are doing something wrong. Since you haven't mentioned stuff like precisely what command you used or precisely what result you got there's no way to tell. Could you take a bug report like: "'lsof' doesn't show the file" and do anything with it?
Re paragraph 2: "exclusive locks" are the only kind that exist. If you want a mandatory lock, read mandatory.txt. But this won't help much. If you do it correctly, you will succeed, and vxftpd will experience failed writes. It will probably detect this and inform the ftp client of the error.
Re paragraph 3: I wonder how the data gets to the file?
I looked at the source code for vsftpd. I see it has a logging option. Turn that on and parse the log to see if the file is finished.
I wonder if this is a vfstpd bug or not. Hard to belive that an open file in UNIX does not have a "flag" soemwhere.
Have you tried /proc/<pid>/fd? That must be of help. Tom
Here is the ouput during the download of the file. You can notice how the size between the first and the last command is different, however, lsof neither fuser has shown any result that indicate that the file is open.
Perhaps I'm not using these commands correctly ... but I've followed the SuSe documentation and appears to be correctly.