Remove Duplicate Lines in File

Shell Programming and Scripting
1

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.

FileA

1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231

and I want to simply it with no duplicate line as file below.

FileA

1253-6856
3101-4011
1827-1356
1822-1157
1000-1410
1822-1231

How can I do that in KSH?

2

Try the following command:

awk '!x[$0]++' file > file.new

Nir

3

Thanks Nir! This is awesome!

4

uniq file > file.new

If the files are not already sorted then:

sort file | uniq > file.new

5 
sort -u filename > filename.new
2 Likes
6

Thanks ssk!

Your solution is so simple and so effective!

Thanks,

Sumit Garg