I need to grep all those lines present in file1 from the other file (file2).
One way will be to run a for loop on file1 and grep in file2. But my data volume is very high an it's taking 5-6 hours.
Can you please suggest the fastest way to achieve this (may be using awk/sed)
Thanks Guru for your prompt response
But my 2nd file size is 15 GB, and the 1st file size is 5 GB. So just wanted to know can this proces be made faster.
I was also was wondering if the lines (as obtained from file2) can be arranged as per the search lines present in file1.
grep, sed, and awk would all do the same thing: read the first file line by line, and check the second file for occurrences each time, chugging through approximately 75 GB (15*5) of data.
One way it could be done faster would be a script/program that reads the second file (which looks like the wanted information is in the same place on every line), creates a hash/list of the numbers and according line numbers, and the only has to go through the first file once.
First question is does this absolutely need to be faster? How many times are you going to run it? If it's a single-shot, then perhaps just letting it run to completion is the best solution.
Secondly, the first file looks like it is a sequence. If so, then perhaps a regular expression could be used rather than a volume of 5 GB of memory. If not a regular expression, then possibly a code that determines if a piece of the line matches the base + the sequence -- an arithmetic operation, which might be faster than string comparisons (for example, some mainframes & supercomputers had multiple units for arithmetic).
Thirdly, if you have sufficient IO throughput as well as multiple cores, then one could write a program that internally divides the main file into pieces by keeping track of start-stop line positions, and then uses processes or threads to process one segment each. A less elegant solution along the same lines would be to spilt the files into n sections, each in a file, and then run n instances of grep.
Fourthly, splitting the task up among a network of machines that might share the disk; as well as the easiest (but not cheapest) solution: get a faster box.
Assuming both files are sorted, maybe you can use "join".
If all the 300 million numbers of file1 start with 372846 (if not, then multiple passes maybe), then you can treat them as integers (minus the prefix). This way you can store them as bitmaps and do look up of the numbers (check prefix first separately) from file2. The first chapter of Jon Bentley's book "programming pearl" talked exactly about this problem.
