How to remove duplicates without sorting

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hello,

I can remove duplicate entries in a file by:

sort File1 | uniq > File2

but how can I remove duplicates without sorting the file?

I tried cat File1 | uniq > File2 but it doesn't work

thanks

2

Many ways using awk or perl. Search this site using the duplicates keyword.

3

Before my post I found on an older post

awk ' !x[$0]++' Infile > outFile

but it was giving syntax errors when exectued.

Then I changed awk to nawk and it worked.

thanks

4

You don't need a sort and uniq for removing duplicates

same can be accomplished by "-u" option

sort -u filename > new.filename
5

try to use nawk or /usr/xpg4/bin/awk