Remove Duplicate Lines

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hi

I need this output. Thanks.

Input:

TAZ
YET
FOO
FOO
VAK
TAZ
BAR

Output:

YET
VAK
BAR
2

try..

 
sort filename|uniq  -u
3 
awk '{b[$0]=++a[$0]} END{for(i in b){if (b==1) {print i}}}' filename
1 Like
4

Awk solution is great thanks :slight_smile:

5

so what's the reason to set another array b?

awk '{a[$0]++} END{for(i in a){if (a==1) {print i}}}' infile
1 Like
6

@rdcwayx - You are right. Thanks for simplifying.

7

try also:

awk '{b[$0]=1}; a[$0]++ {delete b[$0]} END {for (i in b) print i}' input
8

That solution doesn't work. If $0 appears an odd number of times, it will be printed.

Regards,
Alister

9

I have a problem this solution. I don't want sort the text. they are doing it.

input:

111
222
444
111
333
888
222
666

output:

444
333
888
666
10

Not sure if this is the most efficient use of memory:

awk ' b[$0] {c[$0]++} {a[NR]=$0;b[$0]++} END {for (i=1;i<=NR;i++) if (!(a in c)) print a}' file
444
333
888
666
11 
awk ' NR==FNR {a[$0]++; next} a[$0]==1' file file