How to delete duplicate records based on key

Shell Programming and Scripting
1

For example suppose I have a file which contains data as:
$cat data
800,2
100,9
700,3
100,9
200,8
100,3

Now I want the output as
200,8
700,3
800,2

Key is first three characters, I don't want any reords which are having duplicate keys.

Like sort +0.0 -0.3 data can we use similarly in uniq command?

Actual file contains more than 3 million records so I think any shell script will take lots of processing time. I want some fast command.

Please share your thoughts!

Thanks
Sumit

2 
awk -F "," ' {
  cnt[$1] ++
  sav[$1] = $0
} 
END {
  for (x in sav)
     if (cnt[x] == 1)
       print sav[x]
}' your-file

if you have enough memory, this may works.
maybe the following is useful in memory tensive situation

awk -F "," '
NR == FNR {
  cnt[$1] ++
}
NR != FNR {
  if (cnt[$1] == 1)
    print $0
}' your-file your-file
3

Thanks a lot for your amazing code!

But it worked for sample data I have given.

Your first code is giving following error:
awk: 0602-590 Internal software error in the tostring function on

and second code really worked:
It took 3 min 16 seconds to process 3407871 records :slight_smile:

Really cool! I was breaking my head in sort and uniq command !

Once again thank you!

Regards
Sumit

4 
awk '{ x[substr($0,1,3)]++; y[substr($0,1,3)] = $0 }
END { for ( n in x ) if ( x[n] == 1 ) print y[n] }' data | sort
5

thank you it also worked!

6

But can you guys please explain the codes, so that I can understand what exactly it is doing? I really appreciate ur help!

Regards
Sumit

7

Only second code is working-- first and third one is giving error --
"awk: 0602-590 Internal software error in the tostring function on"

Thanks
sumit

8

Can I get the seond code's output in sorted order.

Thanks
Sumit

9

Perl:

perl -F, -lane'
  $u{$F[0]}++; $r{$F[0]} = $_;
  print join $/, map $r{$_}, 
    grep $u{$_} == 1, 
      sort {$a<=>$b} keys %u if eof
  ' infile
10 
awk -F"," '{
a[$1]++
b[$1]=$2
}
END{
for(i in a)
	if(a==1)
		print i","b
}' file