Perl's buffered I/O is causing me to miss latest log file entries in log colorizer. How to fix?

Shell Programming and Scripting
1

I've been finding myself using a log file colorizer written in perl to reformat and colorize the output from many different programs. Mainly, however, I use it to make the output from "tail -f" commands more readable.

The base perl script I use is based on "colorlogs.pl" available from the resentment dot org web site (I would post the URL, but I am but an egg in these forums and hence not worthy ... I can mail or PM it to anyone who is curious). It's a nice little script, and it works well for busy log files. I've extended it a little, but most of it is as originally written.

Unfortunately, on our test system, the logs I'm viewing aren't very busy, and it could be several minutes .. or hours ... between changes in the file.

Because of the very low level of actitivity, I've discovered that the last several lines aren't always displayed (it wasn't obvious on a system with messages coming in every few seconds), and the culprit appears to be the fact that the perl open and print commands perform buffered I/O. This is surely good most of the time, but not in this instance.

Example calling syntax:

 tail -f somefile | colorlogs.pl

The code is fairly simple ... it has a list of colornames associated with various ANSI control sequences, it reads a config file which associates those colornames with various arbitrary strings to match, and then it reads STDIN to obtain the output text to colorize.

# Parse STDIN
while ($line=<>) {
    $matched = 0;
    foreach $string (keys %config) {
        if ($line =~ /$string/) {
             $otextcolor=$textcolor;
             $textcolor=$config{$string};
             if (($textcolor eq $scolor1)
             or ($textcolor eq $scolor2)
             or ($textcolor eq $scolor3)) {
                 $matched = 1;
             }
         }
     }
    if (!$matched) {
        print "$colors{default}$line";
    } else {
        if ($textcolor eq "cyan") {
                print "$colors{default}===> Incoming\n$colors{cyan}$line$colors{default}";
        } elsif ($textcolor eq "brightcyan") {
                print "$colors{default}<=== Outgoing\n$colors{brightcyan}$line$colors{default}";
        } elsif ($textcolor eq "cyanza") {
                print "$colors{default}*** Special ***\n$colors{cyan}$line$colors{default}";
        } else {
                print "$colors{$textcolor}$line$colors{default}";
        }
    }
}

I've done some searching, read the "Suffering from Buffering" web site, and tried various things like sticking:

my $original_fh = select STDOUT;
$| = 1;
select STDERR;
$| = 1;
select $original_fh;

or

select((select(STDOUT), $|=1)[0]);

in the while loop to make STDOUT a "hot" handle, but that doesn't seem to make any difference. The issue might be STDIN. I dunno. :confused:

I also know that sysread and syswrite are unbuffered statements I could use, but I'm not sure how to adapt the existing code to do that. I'm still a relative Perl newbie ... I can tweak existing scripts to bend them to my will, but I'm still in the process of learning basic concepts.

So... How would you folks fix this? :slight_smile: Am I on the right track with setting $| to 1, or would moving to sysread/syswrite be a better idea?

Or rewrite it in C? :smiley:

Any and all comments welcome. Thanks in advance!

2

You could look into sysopen and sysread. They are lower level I/O functions that work unbuffered (unlike <FILEHANDLE> operator).

sysopen(FILEHANDLE, $name, O_RDONLY) or die $!;