Delete duplicate row

Shell Programming and Scripting
1

Hi all,

how can delete duplicate files in file form, e.g.


$cat file1
aaa  123 234 345 456
bbb  345 345 657 568
ccc   345 768 897 456
aaa  123 234 345 456
ddd  786 784 234 263
ccc   345 768 897 456
aaa  123 234 345 456
ccc   345 768 897 456

then i need ouput file1 some,

$cat file1
aaa  123 234 345 456
bbb  345 345 657 568
ccc   345 768 897 456
ddd  786 784 234 263

Thanks you,

2

Did you consider searching these forums, or even look at the link at the bottom of this page?

1 Like
3

off course but not easy aslo some links is close,

thanks,

4

you couldn't find this one?

sort -u file

Believe the awk solution maintains order,

awk '!seen[$0]++' file
5

It does maintain the order, is quick and dirty.
A more explicit and efficient version is

awk '(!($0 in seen)) {print; seen[$0]}' file1