# Diff two files with threshold value

**URL:** <https://community.unix.com/t/diff-two-files-with-threshold-value/295368>\
**Category:** Shell Programming and Scripting\
**Created:** [September 15, 2011, 12:03pm UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368 "2011-09-15T12:03:18Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nishi2011](https://community.unix.com/letter_avatar/nishi2011/32/6_449c533f586939b6e5f246632a46421f.png) [@Nishi2011](https://community.unix.com/u/Nishi2011)\
**Post date:** [September 15, 2011, 12:03pm UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/1 "2011-09-15T12:03:18Z")

</div>

i have two big file which have thousand of line.  
i have to sort on two key fields then diff the file.

if the interger value of one of the column is less then or greater then 1 it should ignore it.

for example  
File1

```nohighlight
abc|7000|jhon|2.3
xyz|9000|sam|6.7
pqr|8000|kapi|4.6    

```

File2

```nohighlight
abc|7000|jhon|2.3
xyz|9000|sam|6.7
pqr|8000|kapi|4.5
uvw|1000|abe|5.6

```

diff Output --expected

```nohighlight
uvw|1000|abe|5.6

```

the commands i tried out are below

```nohighlight
sort file1 > sortfile1 |sort file2 sortfile2 | diff sortfile1 sortfile2

```

output

```nohighlight
2c2,3
< pqr|8000|kapi|4.6
---
> pqr|8000|kapi|4.5
> uvw|1000|abe|5.6

```

---

<div class="post-metadata">

**Author:** ![birei](https://community.unix.com/letter_avatar/birei/32/6_449c533f586939b6e5f246632a46421f.png) [@birei](https://community.unix.com/u/birei)\
**Post date:** [September 15, 2011, 3:45pm UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/2 "2011-09-15T15:45:41Z")

</div>

Hi,

Your explanation is not clear for me:

> [@](#):
>
> i have two big file which have thousand of line.

Both files have same number of lines?

> [@](#):
>
> i have to sort on two key fields then diff the file.

Which fields?

> [@](#):
>
> if the interger value of one of the column is less then or greater then 1 it should ignore it.

I don't understand this.

> [@](#):
>
> diff Output --expected
> 
> ```plaintext
> uvw|1000|abe|5.6
> 
> ```

I don't understand that output. Lines that exists in one file but not in the other?

Regards,  
Birei

---

<div class="post-metadata">

**Author:** ![Nishi2011](https://community.unix.com/letter_avatar/nishi2011/32/6_449c533f586939b6e5f246632a46421f.png) [@Nishi2011](https://community.unix.com/u/Nishi2011)\
**Post date:** [September 16, 2011, 1:30am UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/3 "2011-09-16T01:30:25Z")

</div>

Hi thanks for your response

1) both the file have different number of line  
file1 may have 50 thousand records and file 2 may have 51 thousand record

2) I have to sort the file on the second and third column  
for example  
i have to sort first file1 and file2 on column 7000 and column jhon

```nohighlight
abc|7000|jhon|2.3
xyz|9000|sam|6.7
pqr|8000|kapi|4.6

```

3) the file 1 have

```nohighlight
pqr|8000|kapi|4.6

```

file 2 have

```nohighlight
pqr|8000|kapi|4.5

```

[FONT=monospace]  
there is difference of .1 in forth column (4.6 and 4.5)  
the diff should ignore if the difference is of .1 in the forth column

4) if the difference is of more then .1 it should report in output

---

<div class="post-metadata">

**Author:** ![birei](https://community.unix.com/letter_avatar/birei/32/6_449c533f586939b6e5f246632a46421f.png) [@birei](https://community.unix.com/u/birei)\
**Post date:** [September 16, 2011, 4:17am UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/4 "2011-09-16T04:17:48Z")

</div>

Therefore, the output should be:

1.- Lines that exists only in one of the two files.  
2.- Lines whose first three columns are the same but the difference between the number of the fourth is different from 0.1 or -0.1. In that case, the line of what file is what you have to write to output?

Regards,  
Birei

---

<div class="post-metadata">

**Author:** ![Nishi2011](https://community.unix.com/letter_avatar/nishi2011/32/6_449c533f586939b6e5f246632a46421f.png) [@Nishi2011](https://community.unix.com/u/Nishi2011)\
**Post date:** [September 16, 2011, 5:22am UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/5 "2011-09-16T05:22:29Z")

</div>

Thanks Birei for Quick response  
Yes you are correct

Fourth column is Real number and if its difference is 0.1 or -0.1 it is acceptable so no need to be in output.

If the difference is more then 0.1 it should be in output.  
let me put it in example

1) file1

```nohighlight
abc|7000|jhon|2.3
xyz|9000|sam|6.7
pqr|8000|kapi|4.6
lmn|3000|kapi|4.6

```

2) file 2

```nohighlight
abc|7000|jhon|2.3
xyz|9000|sam|6.7
pqr|8000|kapi|4.5
lmn|3000|kapi|4.1

```

Expectation  
1)the third row is having diff of 0.1 so it is acceptable, not required to be in output

2) the forth row is having a diff more then 0.1 it is 0.5 so should be in the output.  
Output

```nohighlight
> lmn|3000|kapi|4.1
< lmn|3000|kapi|4.6

```

---

<div class="post-metadata">

**Author:** ![spynappels](https://community.unix.com/letter_avatar/spynappels/32/6_449c533f586939b6e5f246632a46421f.png) [@spynappels](https://community.unix.com/u/spynappels)\
**Post date:** [September 16, 2011, 7:35am UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/6 "2011-09-16T07:35:45Z")

</div>

do you want output if the file occurs in only 1 file, or only if it appears in both files and has a difference of more than 0.1 in column 3?

---

<div class="post-metadata">

**Author:** ![Nishi2011](https://community.unix.com/letter_avatar/nishi2011/32/6_449c533f586939b6e5f246632a46421f.png) [@Nishi2011](https://community.unix.com/u/Nishi2011)\
**Post date:** [September 16, 2011, 7:58am UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/7 "2011-09-16T07:58:50Z")

</div>

yes  
only if it appears in both files

---

<div class="post-metadata">

**Author:** ![birei](https://community.unix.com/letter_avatar/birei/32/6_449c533f586939b6e5f246632a46421f.png) [@birei](https://community.unix.com/u/birei)\
**Post date:** [September 16, 2011, 8:43am UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/8 "2011-09-16T08:43:38Z")

</div>

Try next 'Perl' script:

```nohighlight
$ cat file1
abc|7000|jhon|2.3
xyz|9000|sam|6.7
pqr|8000|kapi|4.6
lmn|3000|kapi|4.6
$ cat file2
abc|7000|jhon|2.3
xyz|9000|sam|6.7
pqr|8000|kapi|4.5
lmn|3000|kapi|4.1
$ cat script.pl
use warnings;
use strict;

@ARGV == 2 or die qq[Usage: perl $0 file1 file2\n];

my $num_file = 1;
my %line;

while ( <> ) {
        chomp;
        my @f = split /\|/;

        if ( $num_file == 1 ) {
                $line{ "@f[0..$#f-1]" } = $f[-1];
                next;
        }

        if ( $num_file == 2 ) {
                if ( exists $line{ "@f[0..$#f-1]" } and abs( $line{ "@f[0..$#f-1]" } - $f[-1] ) > 0.1 ) {
                        printf "> %s\n< %s\n", 
                                join( "|", @f[0..$#f-1], $line{ "@f[0..$#f-1]" } ),
                                join( "|", @f[0..$#f-1], $f[-1] );
                }
        }
} continue {
        ++$num_file if eof;
}
$ perl script.pl
Usage: perl script.pl file1 file2
$ perl script.pl file1 file2
> lmn|3000|kapi|4.6
< lmn|3000|kapi|4.1

```

Regards,  
Birei

---

<div class="post-metadata">

**Author:** ![Nishi2011](https://community.unix.com/letter_avatar/nishi2011/32/6_449c533f586939b6e5f246632a46421f.png) [@Nishi2011](https://community.unix.com/u/Nishi2011)\
**Post date:** [September 16, 2011, 11:03am UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/9 "2011-09-16T11:03:40Z")

</div>

It work perfectly for forth column

But it don't catch the difference in 1 to 3 column

for example  
file1  
pqr|7000|jhon|2.3  
xyz|9000|sam|6.7  
pqr|8000|kapi|4.5  
lmn|3000|kapi|4.6

file 2

pqr|7000|jhon|2.3  
xyz|12000|sam|6.7  
pqr|8000|kapi|4.5  
lmn|3000|kapi|4.1

if the the second row second column is different it don't work  
file1  
xyz|9000|sam|6.7

file2  
xyz|12000|sam|6.7

---

<div class="post-metadata">

**Author:** ![birei](https://community.unix.com/letter_avatar/birei/32/6_449c533f586939b6e5f246632a46421f.png) [@birei](https://community.unix.com/u/birei)\
**Post date:** [September 16, 2011, 12:09pm UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/10 "2011-09-16T12:09:56Z")

</div>

Seven posts to try to make clear what you are trying to get, and still confused. I think we are not understanding each other, and may be my problem. I will quote:

You said:

> [@](#):
>
> 2) I have to sort the file on the second and third column

To question of spynappels:

> [@](#):
>
> do you want output if the file occurs in only 1 file, or only if it appears in both files and has a difference of more than 0.1 in column 3?

your answer is:

> [@](#):
>
> yes  
> only if it appears in both files

The program compares the first three fields of each line in both files. Only if they are the same compares the fourth column. In your last example the second field is different (9000 \< 12000). For the program those lines are different so there is nothing to compare and nothing to send to output.

You post different input with each message. Please, try to help me a little and tell what 'same line in both files' means to you. Say what to do when field1, or field2, or field3 is different in both files. Say what to do if there is a line in one file that doesn't exists in the other one (all fields different) and post an example of input file with all cases and exactly the expected output.

Regards,  
Birei.

---

<div class="post-metadata">

**Author:** ![Nishi2011](https://community.unix.com/letter_avatar/nishi2011/32/6_449c533f586939b6e5f246632a46421f.png) [@Nishi2011](https://community.unix.com/u/Nishi2011)\
**Post date:** [September 16, 2011, 1:52pm UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/11 "2011-09-16T13:52:58Z")

</div>

Birei sorry for not being clear,

the script work fine.  
Perl is new to me so was not able to translate the logic.  
but this script works fine for me 🙂

Thanks Birei for all your effort

---

<div class="post-metadata">

**Author:** ![birei](https://community.unix.com/letter_avatar/birei/32/6_449c533f586939b6e5f246632a46421f.png) [@birei](https://community.unix.com/u/birei)\
**Post date:** [September 16, 2011, 4:30pm UTC](https://community.unix.com/t/diff-two-files-with-threshold-value/295368/12 "2011-09-16T16:30:35Z")

</div>

It does not matter.

You can safely indicate if the program doesn't work well, but understand that sometimes it is inconvenient to change the code because requirements are not clear from the beginning.

Regards,  
Birei