Remove duplicates from comma separated list

UNIX for Beginners Q & A
1

Hi,

I have following input file:

niki      niki
niki1    niki
niki2    niki,niki2
niki3    niki,niki3,niki
niki4    niki4,blabla
niki5    jkjkl
niki6    niki60,niki6

I would like to delete lines with identical matches completely and remove the selfmatches in the other lines.

Expected output:

niki1    niki
niki2    niki 
niki3    niki,niki
niki4    blabla
niki5    jkjkl
niki6    niki60

Is that somehow possible?

I tried several things with awk but I can not find a way to solve it properly.

I hope someone can help here.

Thanks!

2

You could use split() to put $2 in an array and then step thru it like this:

awk '
{ 
   a=""
   e=split($2, vals, ",")
   for(i=1;i<=e;i++) if(vals!=$1) a=a","vals
   if(a) print $1,substr(a,2)
} ' infile
3

Try also

awk '
        {gsub ($1 "(,|$)", _, $2)
         gsub (/,,/, ",", $2)
         gsub (/^,|,$/, _, $2)
         $1 = $1
        }
$2
' file