How to process only new files

Shell Programming and Scripting
1

Dear Experts,

I desperately need help in some scripting that im totally stuck at.

I have some files like this :-

-rw-rw---- 1 rtp99 ticgroup 2603099 Jul 17 13:24 cft.CO0102.20070717051933.20070717052451
-rw-rw---- 1 rtp99 ticgroup 3040772 Jul 17 13:24 cft.CO0101.20070717051133.20070717052453
-rw-rw---- 1 rtp99 ticgroup 3102533 Jul 17 13:36 cft.CO0102.20070717052451.20070717053611
-rw-rw---- 1 rtp99 ticgroup 3069764 Jul 17 13:37 cft.CO0101.20070717052453.20070717053741
-rw-rw---- 1 rtp99 ticgroup 3159779 Jul 17 13:48 cft.CO0102.20070717053611.20070717054803
-rw-rw---- 1 rtp99 ticgroup 3167636 Jul 17 13:48 cft.CO0101.20070717053741.20070717054844
-rw-rw---- 1 rtp99 ticgroup 3074107 Jul 17 13:59 cft.CO0102.20070717054803.20070717055925
-rw-rw---- 1 rtp99 ticgroup 3098716 Jul 17 14:00 cft.CO0101.20070717054844.20070717060030
-rw-rw---- 1 rtp99 ticgroup 3182712 Jul 17 14:10 cft.CO0102.20070717055925.20070717061033
-rw-rw---- 1 rtp99 ticgroup 3195226 Jul 17 14:11 cft.CO0101.20070717060030.20070717061148

As you can see the files come in every 2-10 mins. I need to process some info inside these files and dump the info elsewhere. The problem is, i dont know how to only pick up the next new file (latest file), and process it. I tried dumping the output of "ls" onto a file, and then using "comm" to find the new file, but this method seems to be slowing down my server a lot.

Appreciate any help. Many thanks.

Sara

2

To find the latest file in the path

ls -t1 cft.C* | head -1 | read filename

So now the variable filename will have the latest file name.

3

you can try this logic

while :
do
ls -lrt | tail -1  
echo # process them what ever you want
sleep 10
done

it takes last file only since i used "tail -1" command

some friends may come with better logic

4

Sorry guys, I forgot to mention that I am only allowed to let the cron run every every 15 mins, so "ls -ltr | tail -1" is might miss out a file. Any way i will be able to pick up only the new files that were not processed the last time cron ran?

5

Method 1: use find -newer and a reference file.

find . -type f -newer ref_file -print |\
while read filename
do
    :  something with $filename
done
touch ref_file

Method 2: check that you haven't processed the file before.

set -- *
for filename
do
    if ! grep -q $filename ref_file
    then
        :  something with $filename
        echo $filename >> ref_file
    fi
done

Combine methods if necessary.

6

Thanks Ygor. I found that helpful. Thanks everyone.