How to FTP all newly created but the current open file?

UNIX for Advanced & Expert Users
1

An application running on HP-UX constantly generates new text log files ( I think using logpipe ). Any new file created requires to be ftp'ed to an offline server, however I want to make sure that the current file being written should not be transferred.
For examples consider the following files on HP-UX server that are transferred to an offline server for processing.

file1 - old already transferred at T1
file2 - old already transferred at T1
file3 - old already transferred at T1
file4 - new needs to be transferred at T2
file5 - new needs to be transferred at T2
file6 - new needs to be transferred at T2
file7 - new needs to be transferred at T2
file8 - new needs to be transferred at T2
file9 - file currently open by application for writing ( don't include this file for transfer at T2, this will be transferred at T3 when the file is close)

Also the script that ftp's the files run on the remote offline server. Is there a command that can be run from remote server that will help identify all the new files but prevent the current open file from getting transferred?

thanks

2

I doubt whether UNIX has built in commands for these requriements. I see two queries here.

1) 'Already FTPed files' not be FTPed again :- Two solutions 1) Record the timings of when ever FTP command runs. Then always pick the files later than last run FTP command. 2) If remote host has permission to move/rename the files then as soon as they got FTPed move/rename them to some other format other than FTP files format. But in real scenarios it is very unlikely that remote host would be having permissions to move/rename files.

2) Not to pick the current growing file:- Assuming only one file is open for writing at any time, why dont you just leave the latest file (ignore file 'ls -lrt | tail -1')?

1 Like
3

You can check to see if any file has a particular file open using the lsof command.

lsof filename

produces no output if no process has that file open. If you're running on Solaris then you can also use the 'fuser' command in the same way.

Unfortunately these must both be run on the local server. So you must run them there or via 'trusted' rsh/ssh connections.

1 Like
4

You could always use a "workaround" like this:

ls -lrt | tail -2 | head -1 | awk '{ print $9 }'

to find out the second oldest filename in a directory and FTP that file, leaving the currently open file alone.

Assuming no other files are written there, use "ls -lrt name*" as a filter otherwise.
Also assuming the logs are rotated at known intervals, you can use cron to schedule the FTP-ing.

1 Like