How to find duplicate line in Linux?

Shell Programming and Scripting
1

Hi, Gurus,

I need find the duplicate record in unix file.

what command I should use for this.

Thanks in advance

2

try:

awk 'a[$0]++' infile
1 Like
3

Thanks for your quick reply.

it works perfect.

Thanks again

4

It works like a charm for me as well. Would you mind very much explaining what 'a[$0]++' means?

5

It is an associative array indexed by whole record and the value is post-incremented.

So for first occurrence of each record the value will be zero due to post-increment. But for next occurrence it will be a non-zero value.

A non-zero value is evaluated as true, hence the default awk action is to print that record.

Your can run below program to understand what is going on:

awk '{ print $0, a[$0]++ }' file