Using an awk script to identify dupes in two files

Shell Programming and Scripting
1

Hello,

I have two files. File1 or the master file contains two columns separated by a delimiter:

a=b
b=d
e=f
g=h

File 2 which is the file to be processed has only a single column

a
h
c
b

What I need is an awk script to identify unique names from file 2 which are not found in the master file

The desired output would be:

h
c

The master file is huge around 1,50,000 lines
The script I had written bu which did not work satisfactorily is as under:

# Use to match items present in 2 text databases

BEGIN {FS="="}
{
#Master file
if (ARGIND==1) {count[$1]=$2;}    
# Second file
if (ARGIND==2) {count2[$1]=$0;}   

}
END
{

for (i in count) 
#To show matching items. Commented
#if (count2 != "") {print count2} 
To show items that do not match the master database
#if (count2 == "") {print count2} 
}

It does show the uniks but also spews out the non-unique words.
Any help would be most appreciated. Please help since my dictionary work is halted

Many thanks in advance,

GIMLEY

2 
awk -F= 'NR==FNR{O[$1]++;next} !($1 in O)' file1 file2
3

file 2 is also huge?

---------- Post updated at 11:32 AM ---------- Previous update was at 11:23 AM ----------

file1 is big,so here O array is really a big array

1 Like
4

I tested with a file of 3.8million records with each key 12 chars long and it took 11 seconds to run, on a thinkpad laptop:

$ head file1.txt
32129329086
48625634304
22144404670
68186949150
73047198101
75779955958
49331642369
02427749207
60560456186
38039769462

$ wc -l file1.txt
3870483 file1.txt
 
$ time awk -F= 'NR==FNR{O[$1]++;next} !($1 in O)' file1.txt file2
a
h
c
b
real    0m11.093s
user    0m9.827s
sys     0m0.171s
5

Hello,
Many thanks for the solution and also the prompt reply.
The script worked beautifully with the sample data. But with real world data it did not work. Should I have mentioned that the right hand side of the master file is in upper ascii or possibly in Unicode code pages other than Latin 1

aabad=����
aabadaabu=�������
aabadabu=�������
aabbey=�����
aabedali=���Ĥ��
aabedin=������
aabeed=����
aabel=����
aabelia=�������
aabelin=������

and the "slave" file has only lower ascii but no upper ascii

kishor
aabaadabu
aabhar

The output should have been

kishor
aabhar

Any solutions please,

Best regards and sorry for the hassle,

Gimley

6

awk will work if you adjust the locale settings. what does

locale 
locale -a

show right now? locale gives the current setting, locale -a shows the available ones.

1 Like
7

I am sorry to hassle you guys like this. I am on Windows Vista and locale does not work. Basically my locale is ISO 1252 (ANSI - Latin I), which should handle lower as well as upper ASCII characters.
I still am perplexed why the single chars work whereas the longer strings do not work.
Sorry to be such a bother, but this is a real mystery

Gimley

---------- Post updated at 11:57 AM ---------- Previous update was at 11:49 AM ----------

Sorry Guys my goof-up. In my excitement to get the data working, I had forgotten to put the file separator.
BEGIN {FS="="}
NR==FNR{O[$1]++;next} !($1 in O)
It works beautifully and ran through the records like a breeze.
Please excuse my stupidity.
Best regards and many thanks to all who helped me out.
GIMLEY