How can i delete the duplicates based on one column of a line

Shell Programming and Scripting
1

I have my data something like this

(08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
(08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa
(08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts
(08/03/2009 22:57:42.425)(:) Ravi vvvvvvvvvvvvvvvvvvsssssssss bsbbbbs
(08/03/2009 22:57:42.426)(:) John bgbhhhhhhhhhhhhhhhhh dddddddddddddd
(08/03/2009 22:57:42.427)(:) king hhhhhhhhhhhhhssssss rr

Here i need to take the 3rd column as the key foir finding the duplicate rows. I need the output to have the rows with only one king,one john and so on...

Output expected :

(08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
(08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa
(08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts
(08/03/2009 22:57:42.425)(:) Ravi vvvvvvvvvvvvvvvvvvsssssssss bsbbbbs

can some expert help me with this? this will be very helpful for my script.

2

May not be efficient

awk '!arr[$3]++ {print}'  file
3

I am getting syntax error with that command. Could you verify the syntax please?

4

Use nawk or /usr/xpg4/bin/awk on Solaris.

Regards

5

I tried using

nawk '!arr[$3]++ {print}' file

it's not removing the duplicates..just printing all the rows.

6

Very inefficient:

awk '{x = $3
if (x != y) print
y = $3
}' file
7

This is printing all the lines without removing the lines with duplicate column3

8

This should work:

awk '!arr[$3]++' file

This is my output:

$ cat file
(08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
(08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa
(08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts
(08/03/2009 22:57:42.425)(:) Ravi vvvvvvvvvvvvvvvvvvsssssssss bsbbbbs
(08/03/2009 22:57:42.426)(:) John bgbhhhhhhhhhhhhhhhhh dddddddddddddd
(08/03/2009 22:57:42.427)(:) king hhhhhhhhhhhhhssssss rr
$
$ awk '!arr[$3]++' file
(08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
(08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa
(08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts
(08/03/2009 22:57:42.425)(:) Ravi vvvvvvvvvvvvvvvvvvsssssssss bsbbbbs

Regards

9 
awk '{x = x + 1
> y[x] = $3
> for ( z = 0; z <= x; ++z ) {
> if ( z == x )
> print
> if ( y[z] == $3 )
> break
> }}' file1
10 
sort -u -k 3,3 file
11

Hi Franklin,
I am afraid, I tried that in my SunOS box and still getting the syntax error...Not sure if that won't work in Sun box.

12

Use nawk or /usr/xpg4/bin/awk on Solaris.

Regards