remove duplicate

Shell Programming and Scripting
1

i have a text its contain many record, but its written in one line,
i want to remove from that line the duplicate record,
not record have fixed width ex: width = 4
inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc
i want the outputfile =abc cdf fgh
only those records
can any one help me as unix script sh?
thnx in advance.

2

super quick and dirty way is use the space as your seperator and load each element into a hash.

hashes can only contain unique keys. so if you load up a hash then you are sure to get no duplicates.

then you just spit out each key and reform your line.

3

Try...

tr ' ' '\012' < test.txt | sort -u | paste -s -d' ' - > outfile

4

thnx Ygor, Optimus_P
but what if the record contain space
ex: my record size 5
test.txt=ab cdefr tab cdab cdefr tasdfg
the result must be
ab cdefr tasdfg
i try :
dd if=$1 of=$1.temp cbs=5 conv=unblock
sort $1.temp -u >$1.temp1
cat $1.temp1|tr "\n" " ">$1.new ?

5

I don't understand, kazanoova ... wouldn't the output be ab cdefr tab cdab tasdfg?