question about "uniq"

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

I'm sure this is very basic, but I'm just a beginner, so please be kind. :slight_smile:

I think I understand how to "sort" a text file, and how to save the unique lines to a new file:

sort file.txt | uniq > newfile.txt

Now, I want to do something similar, but only the first few entries of the line must be unique in order to be saved to the new file.

I have a text file with data that looks like this:
1.2 2.1 0 1 2 3 4 5 6 7 8 9 11 10
1.2 2.1 0 1 2 3 4 5 6 7 8 10 11 9
1.2 2.1 0 1 2 3 4 5 6 7 8 10 9 11
1.3 3.1 0 1 2 3 4 5 6 7 8 11 10 9
1.2 2.1 0 1 2 3 4 5 6 7 9 8 10 11
1.2 2.1 0 1 2 3 4 5 6 7 9 11 10 8
1.3 3.1 0 1 2 3 4 5 6 7 10 11 8 9
1.3 3.1 1.5 0 1 2 3 4 5 6 7 10 9 8 11
1.4 4.1 0 1 2 3 4 5 6 7 11 10 9 8

It's a series of decimal numbers (1.2 2.1, though there are not necessarily only two decimal numbers per line) followed by 12 integers, always beginning with 0.

What I want to do is to sort all of these (easy!), and then keep only the lines whose series of DECIMAL number series is unique. From this list, there are only two entries here to keep: the line beginning with 1.3 3.1 1.5 (second to last) and the line beginning with 1.4 4.1 (last line).

Do any of you know how to do this? I can unix "sort" the lines, but then to eliminate lines that have partial duplication as I've described above?

Thank you VERY much for your help!
Zac

2

You want to define the sort key - then ask sort to do a unique sort, with sort -u
something like:

# You don't have to sort the file using all the fileds first unless you really want to.
# but this does it
sort infile | sort -n -u -k1,1 -k2,2 -k3,3 -k4,4 > outfile
# without the extra sort
sort -n -u -k1,1 -k2,2 -k3,3 -k4,4 infile > newfile