Help with remove duplicate content

Shell Programming and Scripting
1

Input file

data_1 10 US
data_1 2 US
data_1 5 UK
data_2 20 ENGLAND
data_2 12 KOREA
data_3 4 CHINA
.
.
data_60 123 US
data_60 23 UK
data_60 45 US

Desired output file

data_1 10 US
data_1 5 UK
data_2 20 ENGLAND
data_2 12 KOREA
data_3 4 CHINA
.
.
data_60 123 US
data_60 23 UK

If data content in column 1 and column 3 are exactly the same. Then print out only the first record.
Thanks for any advice.

2 
awk '!a[$1 FS $NF]++' infile
1 Like
3 
$
$
$ cat f20
data_1 10 US
data_1 2 US
data_1 5 UK
data_2 20 ENGLAND
data_2 12 KOREA
data_3 4 CHINA
data_60 123 US
data_60 23 UK
data_60 45 US
$
$
$ perl -lne '/(\S+)(\s+(\S+)){2}$/; defined $x{"$1:$3"} || print; $x{"$1:$3"}++' f20
data_1 10 US
data_1 5 UK
data_2 20 ENGLAND
data_2 12 KOREA
data_3 4 CHINA
data_60 123 US
data_60 23 UK
$
$

tyler_durden

1 Like