Detecting new entries in log files

Shell Programming and Scripting
1

Hello All,

I have a script that checks a log file. This will be checked periodically lets say every 5 minutes. I need to get new entries and process these new entries

What do you think is the best strategy here? all i can think of currently is to backup the old file and do

diff

. But future problem will be once the log increases in size, this method will be slower.

Any suggestion is appreciated.

2

Use stat to get the modification timestamp of the file and store it on a variable. Compare it later to check changes. Of course the filesystem should be consistent with updates. Some filesystem may have no modification timestamp? I don't know. Perhaps.

3

thanks. modification time is a good indication of something was updated. but i need to fetch the new entries and process it which stat cannot do.

4

You could read the log in real time with tail -f if that helps. Beyond this, you may need to keep and old version and compare them. Using diff can be awkward because it adds editing messages if you are to shovel the output into ed so perhaps you would be better to count the lines in your saved version and then get extra lines from the current file, something like:-

cp logfile temp_logfile
old_lines=`grep -c "" old_logfile`
temp_lines=`grep -c "" temp_logfile`
((lines=$new_lines-$old_lines))
tail -n $lines temp_logfile
.... some other processing if you like.....
mv temp_logfile old_logfile

I've added the temporary log file so that messages can still be added to the main logfile as you are working on it without it skewing the output, so it givers you a fixed reference point.

i hope that this might help.

Robin
Liverpool/Blackburn
UK

1 Like
5

ill try your suggestion. thanks!

6

Well basically you'll have to check the file's contents from time to time but using stat as the starting point would help save your disk from much IO.

7

If the logfile have date stamp, you can look at them every 5 minutes and get only the whats new the last 5 min

8

how do i do that?

9

Look at this post

1 Like
10

It depends if the content of the file has a date & time at the start of each record though. It could be much neater that my way if that is the case.

Thanks Jotne, I may use that myself elsewhere. :wink: I'm always happy to learn too. :stuck_out_tongue:

Robin