at which situation uniq command fails.
How to delete/remove duplicate lines in an unix file
What is your input like? Is order important? uniq requires lines in sorted order.
If your input data is small enough to be held in main memory, this may do:
awk '{ if(!arr[$0]) { arr[$0]=1; print; } }' < data
But if your input data is larger than a few handfuls of megs, this is inefficient, you may have to sort it with
sort -u
which will sort it and remove duplicates.