File comparison of huge files

Hi all,
I hope you are well. I am very happy to see your contribution. I am eager to become part of it.

I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below

For clear picture, please see attached image

File 1 File 2
----------------- -------------------
Time sig_name sig_val Time sig_name sig_val

0ns sig1 0 0ns sig1 0
0ns sig2 0 0ns sig2 1
0ns sig3 0 0ns sig3 1
0ns sig4 1 0ns sig4 1
1ns sig1 0 1ns sig1 0
1ns sig2 0 1ns sig2 0
1ns sig3 0 1ns sig3 0
1ns sig4 0 1ns sig4 0
2ns sig1 0 2ns sig1 1
2ns sig2 0 2ns sig2 0
2ns sig3 0 2ns sig3 0
2ns sig4 0 2ns sig4 0
3ns sig1 1 3ns sig1 0
3ns sig2 0 3ns sig2 1
3ns sig3 0 3ns sig3 0
3ns sig4 0 3ns sig4 0

Given the two files in the above format, how can i print out the following table from "efficient" file comparison. Efficiency is required as file size is over 3GB

signal number_of_mismatches time_of_mismatch
------ -------------------- ----------------
sig1 2 2ns, 3ns
sig2 2 0ns, 3ns
sig3 1 0ns
sig4 0

I shall really appreciate your response.

Unless you have enough RAM to process such volume of data on the flight, it would be advisable to load all those data into an DBMS (database management system) for further processing.

Otherwise : filter out duplicate lines and save lines that appear only once in file3

cat file1 file2 | sort | uniq -u >file3

(also see the corresponding -T options of sort command (depending on your OS) for using temporary files and/or directory)

Then process file3 (i didn't test it but maybe something like this :

awk '{N[$2]++;A[$2]=(A[$2]=="")?$1:((A[$2]~$1)?(A[$2]):(A[$2]","$1))}END{print "signal" OFS "num of mismatch" OFS "time of missmatch";for(i in A) {print i OFS N/2 OFS A}}' OFS="\t" file3

---------- Post updated at 07:38 PM ---------- Previous update was at 07:27 PM ----------

Regarding the file3 generation, you could also give a try to something like

mknod /tmp/pipeline p
sort -u </tmp/pipeline >file3 &
cat file1 file2>/tmp/pipeline

Thanks ctsgnb.

Could you please explain why do you need to sort?

Secondly, could you please explain the details of the awk code?

Thirdly please explain why use mknod?

I shall really appreciate.

Kind Regards,

---------- Post updated at 03:35 PM ---------- Previous update was at 03:01 PM ----------

It produces the WRONG output as follows.Note sig3 has 2.5 as number which makes no sense

signal num of mismatch time of mismatch
sig1 3 0ns,1ns,2ns,3ns
sig2 3 0ns,1ns,2ns,3ns
sig3 2.5 0ns,1ns,2ns,3ns
sig4 2 0ns,1ns,2ns,3ns

instead, the following (correct) output should be produced

signal number_of_mismatches time_of_mismatch
------ -------------------- ----------------
sig1 2 2ns, 3ns
sig2 2 0ns, 3ns
sig3 1 0ns
sig4 0

---------- Post updated at 03:56 PM ---------- Previous update was at 03:35 PM ----------

/********************* IT WORKS AS FOLLOWS *************************/

Here is what works. ctsgnb your awk script is awesome it works perfectly. Here is what you need to modify

cat file1 file2 | sort | uniq -u > file3.txt

Then do awk using your script on file3.txt

Thanks a lot.

It's the simplest and most reliable way to tell whether a line's a duplicate or not. sort is actually pretty sophisticated, capable of handling extremely huge files without bogging down. Once it's done so, any duplicate lines will show up all in a row, letting you use uniq to get rid of them.

He's actually making a named pipe. On many systems you can just use mkfifo for that.

A named pipe lets you do this:

mkfifo fifo
cat file > fifo &
cat fifo # the first cat writes to the fifo, the second cat reads from it.
# This is effectively cat file | cat.
# Named pipes can be a handy way to join together programs which
# read from filenames and can't easily handle stdin/stdout communication.

Thanks a million ctsgnb and all others.

I am having problems on large files. I am able to get this thing working on 3GB files but when the file size is 11GB, i get this message

sort: write failed: /tmp/sortZgF9MD: No space left on device

despite the fact that i have good hard disk space

Any solution to this.

Please suggest

Really appreciate your help

Kind Regards,
kaaliakahn

I think ctsgnb already mentioned this in his post, but try to see if the sort on your OS, support the "-T" option.
Also post output from

df -k /tmp

. Looks like you are running out of space in /tmp. So basically 2 options, see if your sort can use -T and specify an alternate location for tmp-files or split your file into smaller pieces and do a sort on the smaller split files.

Hi dude2cool

I have centos 5.5 and i do have -T option with sort. Here is what i get when i do

df -k /tmp

[user@localhost6 vga]$ df -k /tmp/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda5 15872636 8207944 6845380 55% /

What do you suggest?

So you have around 6.5 Gig in / which is where your /tmp is. So you are running out of space doing sort.

Is there any other free space on your hard disk that you can use? If you so, use the -T option. Look it up

[man man sort ("man sort")](man(1) [centos man page] sort&section=0&os="man sort")

Pretty easy. Find a directory/FS where you have ample free space and specify it using -T

-T directory    Specifies the directory in  which  to  place
                     temporary files

.

dude2cool you are awesome. You are really too cool dude.

Thanks a lot.

Kind Regards and Hurray to unix.com