Flag duplicate enteries

Shell Programming and Scripting
1

Morning Unix Guru's,
I have written a script which output the contents of a DB. No the DB is not clever to check this or flag and there is no command for the tool to flag this, which human intervention.

But i need to check to see if there are duplicate enteries in the file output and flag them.

So the example below there is duplicate enteries for actlni02 and allocd01.

How do i flag this?

actlni02              LDN_CAM_UX_PRD
actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
allocd01              LDN_WAT_UX_DEV

Any advice will be greatly appericated.

2

Simple solution.

awk '{a[$1]++;if (a[$1]>1) f="* ";print f$0;f=x}'
actlni02              LDN_CAM_UX_PRD
* actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
* allocd01              LDN_WAT_UX_DEV
3

Thanks.. So simple

4

More complex. This version change color to red on all duplicated lines

awk '{
	a[$1]++
	b[NR]=$0
	}
END {
	for (i=1;i<=NR;i++) {
		split(b,q," +")
		s=(a[q[1]]>1)?"\033[1;31m":"\033[0m"
		print s b "\033[0m"
		}
	}' file


actlni02              LDN_CAM_UX_PRD
actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
allocd01              LDN_CAM_UX_DEV
allocd01              LDN_WAT_UX_DEV

Edit: some shorter

awk '{a[$1]++;b[NR]=$0;c[NR]=$1} END {for (i=1;i<=NR;i++) print ((a[c]>1)?"\033[1;31m":"\033[0m") b "\033[0m"}' file
5

if you need mark all duplicate enteries, try this:

awk 'NR==FNR{a[$1]++;next} a[$1]>1 {$0="* " $0}1' infile infile

* actlni02              LDN_CAM_UX_PRD
* actlni02              LDN_DKTP_PC
actlnp03             LDN_CAM_UX_PRD
adsdbi01             LDN_CAM_UX_PRD
algomd01            LDN_WAT_UX_DEV
* allocd01              LDN_CAM_UX_DEV
* allocd01              LDN_WAT_UX_DEV
6

Nice. It will be much faster to read file twice compare to add file to array than loop trough every element end test.
Here is a color version of rdcwayxs script

awk 'NR==FNR{a[$1]++;next} a[$1]>1 {$0="\033[1;31m" $0 "\033[0m"}1' file file