duplicate line in a text file

Shell Programming and Scripting
1

i would like to scan file in for duplicate lines, and print the duplicates to another file,
oh and it has to be case insensitive.

example

line1
line2
line2
line3
line4
line4

outputfile:
line2
line4

any ideas

2 
perl -ne '$l = lc(); print if $m{$l}++; ' file

lc returns the specified string in lowercase; in this case, with no parameter, it defaults to the current input line. $m{$l} is the count of number of times we have seen $l; if it's nonzero, it's a duplicate, so we print it.

3

thanks it's worked for one file, but i forgot to add that if i need it to run on a directory and make sure to do in on a file only and not subdirectory

what modifications would you do

4

Loop over files in a directory, and remove duplicates? Do you want to replace the files?

for f in directory/*; do
  test -d "$f" && continue   # skip if it's a subdirectory
  perl -ne '$l = lc(); print if $m{$l}++; ' "$f" >"$f.tmp"
  mv "$f.tmp" "$f"
done
5

Thank you Era, it worked, :slight_smile:

6 
awk '{
  a[$0]++
}
END{
 for (i in a)
 if (a=2)
 print i
}' filename