compare the similar files

Shell Programming and Scripting
1

I got many pair files, which only have small difference, such as more space, or more empty line, and some unreadable characters.

If list by commend "diff", I can see many many difference.

So I'd like to write a script to compare the pair files, if 95% contents are same, I will think they are similar.

Any suggestion for it?

2

you can use

diff -w file1 file2

to ignore all the whitespace differences.
That will not take care of the unprintable characters though.

3

Hi.

I would normalize the files first: remove empty lines, squeeze multiple spaces to one, remove the unprintable characters, etc., and then compare the files.

You may also be able to get some guidelines from The software and text similarity tester SIM

Good luck ... cheers, drl