Remove duplicate

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hi,

How can I replace || with space and then remove duplicate from following text?

T111||T222||T444||T222||T555

Thanks in advance

2

what have you tried?

3

save the text in a file e.g. 'file.txt':
input: file.txt

T111||T222||T444||T222||T555

command:

cat file.txt |tr -s '||' '\n'|sort|uniq -u|tr -s '\n' ' '

output:

T111 T444 T555
4

What duplicate you want to delete
please show how your output shall looks like

5

tinku981,
check this out:

$ awk -F"[|][|]" '{for(i=1;i<=NF;i++) print $i }' file|sort|uniq -c
   1 T111
   2 T222
   1 T444
   1 T555
6

Hello tinku981,

Could you please try following. Lets say file remove_desired_char have the all information.

cat remove_duplicate_char | tr "||" "\n" | awk '!a[$NF]++' | tr "\n" " "

Output will be as follows.

T111  T222 T444 T555

Thanks,
R. Singh

7

There's most likely a faster way, but this should do the trick:
echo "T111||T222||T444||T222||T555" |sed 's/||/\n/g' |sort -u |awk '{ printf "%s ", $0 }'

8 
awk -F"||" '{for(i=1;i<=NF;i++) if (!a[$i]++) {str=str" "$i}} END{print str}' infile

If you have multiple lines of data, they will be merged together and duplicates will be removed.

9

Hello,

Could you please try the following code and let me know if this helps.
Lets say file named remove_duplicate_char have all values mentioned by you.

cat remove_duplicate_char | tr "|" "\n" | awk '!a[$NF]++' | tr "\n" " "

Output will be as follows.

T111  T222 T444 T555

Thanks,
R. Singh

10

Another approach:

awk '
        {
                gsub ( /\|\|/, " " )
                for ( i = 1; i <= NF; i++ )
                {
                        if ( !( $i in A ) )
                                printf "%s ", $i
                        A[$i]
                }
                printf "\n"
        }
' file
11

The latter can be optimized

awk '
BEGIN {FS="[|][|]"}
{
  for (i=1;i<=NF;i++) {
    if (A[$i]++==0) {printf "%s", sep $i; sep=" "}
  }
  print ""
}
' file

GNU awk even takes

awk '
BEGIN {RS="[|][|]"}
A[$1]++==0 {printf "%s", sep $1; sep=" "}
END {print ""}
' file