Perl for comparing numbers from previous lines in a file?

Shell Programming and Scripting
1

Hi everyone

I have a question for you, as I am trying to learn more about Perl and work with some weather data. I have an ascii file (shown below) that has 10 lines with different columns. What I would like is have Perl find an "anomalous" value by comparing a field with the values from the last 3 lines (for the same field, of course). For instance, if we look at the data below:

A15 26.62 765  27.30 4.3
A11 26.63 763  27.28 4.2
A12 26.68 767  27.29 4.3
A16 26.64 768  27.30 4.2
A11 26.62 761  27.31 4.1
A15 26.62 765  27.30 4.3
A15 26.63 763  27.28 4.2
A16 26.68 767  2.29 4.3
A17 26.64 768  27.30 4.2
A18 26.62 761  27.31 4.1

We see here that on the 8th line, the value in column 4 is "off" with respect to the previous lines (all of which are approx. 27.3 degrees Celsius).

Thus, how can I do to compare the values for a line in column 4 with the values from the previous 3 lines in order to detect these "bad" data values? Assume that any value that dirfts +/- than 0.5deg C is "bad data"

I am not sure how to proceed. Do I need to load the data into an array? Do I need to read the file line by line? Or maybe something else?

Any help, suggestions, and especially examples would be really helpful.

Thanks in advance :slight_smile:

Sorry... I forgot to mention that this is what I have in my script so far:

my $file = "$WORK_DIR/tmpdata.asc";

open my $info, $file or die "Could not open $file: $!";
my %line;
local @ARGV = ($file);

while ( <$info> ) {
        my $x1 = $_;
        my @cols = split(" ", $x1);

        print "@cols\n";

}

close $info;
2

How about:

my $file = "$WORK_DIR/tmpdata.asc";

open my $info, $file or die "Could not open $file: $!";
my %line;
local @ARGV = ($file);
my @temps;

while ( <$info> ) {
        my $x1 = $_;
        my @cols = split(" ", $x1);

        if(@temps &&
            abs(@cols[3] - eval(join("+", @temps)) / @temps) > 0.5) {
            print "@cols\n"
        } else {
            shift(@temps) if (@temps > 3);
            push(@temps, @cols[3]);
        }
}

close $info;
2 Likes
3

This works great... thanks Chubler!!