Eliminate redundant data pairs

Shell Programming and Scripting
1

Hello Experts:

I appeal to you to see if you can help me with a small problem. I have a .log file where there is data in two columns (separated by a space).

The file is thus:

0.0 3
0.0 6
0.0 6
0.0 6
0.0 7
0.0 7
0.0 7
0.0 7
0.0 11
0.0 11
0.0 11
0.0 11
0.0 11
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.2 17
0.2 17
0.2 17
0.2 17
0.2 17
0.2 17
0.2 17
0.2 17
0.2 17
0.2 17

The overall length is close to the 20000 rows. I was wondering how eliminate redundant data pairs, for example, from the list above, there are many:

0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17
0.1 17

And the truth is that I need only 1 of those pairs of values. I hope you can help me. Of course, I'll be very grateful.

Cordial Greetings friends.

2

Hello.

Per our forum rules, all posts must be in English.

We do provide translation services for posts from English to a number of languages as a benefit to users. However, posts must be in English.

Please repost in English.

Thank you for your cooperation.

The UNIX and Linux Forums.

3

Please forgive if any translation is mistaken - simply a way to let someone know in their language of our message to them.

Hola.
Por nuestro foro gobierna, todos postes deben ser en ingl�s.
Proporcionamos traducci�n servicios para postes de ingl�s a varios idiomas como un beneficio a usuarios.
Sin embargo, los postes deben ser en ingl�s.
Por favor repost en ingl�s.
Gracias por su cooperaci�n.
El UNIX y Foros de Linux.

4

Eliminating redundant data that is already sorted is easy with 'uniq'

uniq inputfile outputfile
5

If original data is not sorted, the uniq command will give strange results. You may need to sort the data prior to a uniq command.
I manually placed a few of the last entries at the beginning of the file, and see the results:

>uniq uniqsample.txt
0.2 17
0.0 3
0.0 6
0.0 7
0.0 11
0.1 17
0.2 17
6

A frequent asked question, if the file is not sorted:

awk '!a[$0]++' file
7

thank you very much!

Now I have my vectors as I needed.

thanks a lot