I have a script that checks a log file. This will be checked periodically lets say every 5 minutes. I need to get new entries and process these new entries
What do you think is the best strategy here? all i can think of currently is to backup the old file and do
diff
. But future problem will be once the log increases in size, this method will be slower.
Use stat to get the modification timestamp of the file and store it on a variable. Compare it later to check changes. Of course the filesystem should be consistent with updates. Some filesystem may have no modification timestamp? I don't know. Perhaps.
You could read the log in real time with tail -f if that helps. Beyond this, you may need to keep and old version and compare them. Using diff can be awkward because it adds editing messages if you are to shovel the output into ed so perhaps you would be better to count the lines in your saved version and then get extra lines from the current file, something like:-
cp logfile temp_logfile
old_lines=`grep -c "" old_logfile`
temp_lines=`grep -c "" temp_logfile`
((lines=$new_lines-$old_lines))
tail -n $lines temp_logfile
.... some other processing if you like.....
mv temp_logfile old_logfile
I've added the temporary log file so that messages can still be added to the main logfile as you are working on it without it skewing the output, so it givers you a fixed reference point.