removal by substring

Hi guys,

I am trying to remove lines that have a duplicate substring from any part in the file. So, for ex:

433043950359.3 5033 305935 2 2dd 5ffgs DOG43453552.A
3443565634 95460 3435 45 23d 56ggh DOG343211
3423895702359 34 66699 9455 2324 DOG43453552.B

This is a very large file, and I dont think

sort -u filename

will work because I need to look for a substring not an entire string.

any ideas?

You need to qualify what you mean by "substring". For in your example, all three lines contain "3". And "4". And "D".