gref -f taking long time for big file

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

grep -f taking long time to compare for big files, any alternate for fast check

I am using grep -f file1 file2 to check - to ckeck dups/common rows prsents. But my files contains file1 contains 5gb and file 2 contains 50 mb and its taking such a long time to compare the files.
Do we have any alternate for fast/quick check comparetion ?

2

try 'fgrep'

User Commands                                            fgrep(1)



NAME
     fgrep - search a file for a fixed-character string
3

Even fgrep won't scale for larger files as the operation is o(n2). Best approach is to hash one of the files and do a o(1) lookup best case with the other file

4

I want to check/compare file1 rows into file2 rows, if file1 rows present in file2 then needs to display like that compare for all rows in file1 to file 2 and it varies the content also. how to do ? the current one grep -f file1 file2 is working my scenario but its looking such a long time, any alternate ?

5

Can you please post some sample input and output data?

6

Or read the man page on comm - that is designed to find rows found in two files.

7

file1.txt conains:
1 2 3 4 5
1 1 1 1 1
2 2 2 2 2
1 3 5 7 9
4 4 4 4 4

file2.txt contains:
1 2 3 4 5
1 3 5 7 9
7 7 7 7 7
9 9 9 99

the output of file3.txt should be
1 2 3 4 5
1 3 5 7 9

Here grep -f file1.txt file2.txt > file3.txt : its working fine for small size of files but large files size (file size conatns GB) then its not working/not at all comming back...?

8

Try awk - note the smallest file MUST get processed first in this case file1

awk 'FILENAME=="file1" {arr[$0]++}
       FILENAME=="file2" {if($0 in arr) {print $0} }' file1 file2 > duplicates
9

When I use your mentioned command and getting below error,

awk: Internal software error in the symbol table at XXXXXXXXX XXXXXXXXXXX XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXX .
The input line number is 3.81417e+06. The file is file2.
The source line number is 1.

here XXXXXX - row context

any idea / alternate?

10 
sort file1.txt file2.txt |uniq -d

Add "-T /path/to/big/space" if your temp dir is not big enough.

11

Assuming you mean the awk snippet:

You did read in file1 first? That would allocate 50MB of memory.

What OS & version are you on? Some older tools cannot read files larger than about 4GB.