Remove duplicates from a file

Shell Programming and Scripting
1

Hi,

I need to remove duplicates from a file. The file will be like this

0003 10101 20100120 abcdefghi
0003 10101 20100121 abcdefghi
0003 10101 20100122 abcdefghi
0003 10102 20100120 abcdefghi
0003 10103 20100120 abcdefghi
0003 10103 20100121 abcdefghi

Here if the first colum and second column is repeating i need to pick the first record. If not repeating i need to pick the record.

The output shd be.

0003 10101 20100120 abcdefghi
0003 10102 20100120 abcdefghi
0003 10103 20100120 abcdefghi

Thanks in advance for the help. The script could be in Perl or Unix.

2

If the values are ordered, their format is fixed and the uniq implementation on your platform supports the w option:

uniq -w10 infile

Otherwise use awk:

awk '!_[$1,$2]++' infile

On Solaris you should use gawk, nawk or /usr/xpg4/bin/awk.

Or Perl:

perl -ane'print unless $_{$F[0], $F[1]}++' infile
3 
 sort -u -k1,2 infile
4

Hello,
i found this code on a web page which is said to be valid for only gnu linux and delete all lines except duplicate ones, i hope it works (sorry im using solaris10, couldnt try)

# delete all lines except duplicate lines (emulates "uniq -d").

sed '$!N; s/^\(.*\)\n\1$/\1/; t; D' infile
5

Could you explain the perl code please?

6

Yes,
first of all, the code is wrong :slight_smile:
It should be:

unless $_{$F[0],$F[1]}++

... not:

unless $_{@F[0..1]}++

So the script becomes:

perl -ane'print unless $_{$F[0],$F[1]}++' infile

First the command line switches:

So we have the input file read line by line and the @F array automatically populated.

print unless $_{$F[0],$F[1]}++

Print the current record unless the expression ${$F[0],$F[1]}++ returns true in boolean context. We build the hash % whose keys (the concatenation of the first two fields with the subscript separator) are associated with auto-incremented integers. When we see a given key ($F[0] $; $F[1] - the first and the second fields) for the first time, because of the post-incrementing (k++ and not ++k ) its value is 0, i.e. false, so it prints the record.

This will make the concept clear:

% perl -lane'print  $_ ," -> ", $_{$F[0],$F[1]}++' infile
0003 10101 20100120 abcdefghi -> 0
0003 10101 20100121 abcdefghi -> 1
0003 10101 20100122 abcdefghi -> 2
0003 10102 20100120 abcdefghi -> 0
0003 10103 20100120 abcdefghi -> 0
0003 10103 20100121 abcdefghi -> 1

We want only the records with value 0.

Hope this helps.

7

another approach in perl:-

perl -ane ' ! $_{$F[0],$F[1]}++ and print ' infile.txt

perl -ane ' ! $_{$F[0],$F[1]}++ && print '  infile.txt

perl -ane ' print while ! $_{$F[0],$F[1]}++ '  infile.txt

;);):wink: