How to remove duplicates from the .dat file

Shell Programming and Scripting
1

All,

I have a file 1181CUSTOMER-L061411_003500.dat.Z having duplicate records in it.

bash-2.05$ zcat 1181CUSTOMER-L061411_003500.dat.Z|grep "90876251S"
 
90876251S|ABG, AN ADAYANA COMPANY|3550 DEPAUW BLVD|||US|IN|INDIANAPOLIS||DAL|46268||||||GEN|||||||USD|||ABG, AN ADAYANA COMPANY|||||||||Xerox Duns XNAC Link|Researched Legal Name||0|||1181|051726412|125051347
 
90876251S|ABG, AN ADAYANA COMPANY|3550 DEPAUW BLVD|||US|IN|INDIANAPOLIS||DAL|46268||||||GEN|||||||USD|||ABG, AN ADAYANA COMPANY|||||||||Xerox Duns XNAC Link|Researched Legal Name||0|||1181|051726412|125051347

my query is how to remove the duplicate record from a file with the help of a Unix command. Here I have to remove the second data from the file.

Regards
Oracle User

2

If it's ok to lose the original order then

sort -u < YOURFILE

---

Oh, i didn't understand question.

3

Similar questions are asked every week. Look for solution in forum first should be better way.

awk -F \| '!a[$1]++' infile
4

we cannot change the order.

Regards
Oracle User