Remove Duplicate Records

Shell Programming and Scripting
1

Hi frinds,
Need your help.

item , color ,desc
====  ======= ====
1,red ,abc
1,red , a b c
2,blue,x
3,black,y
4,brown,xv
4,brown,x v
4,brown,  x  v

I have to elemnet the duplicate rows on the basis of item.

the final out put will be

 
1,red ,abc
2,blue,x
3,black,y
4,brown,xv

in duplicate records pick the first row.
kindly help me ,how to make it in unix shell .

Thanks in advance

2 
awk -F "," 'a[$1,$2]=="" {print $0 ;a[$1,$2]=$3}' urfile
3

Not as graceful as awk but it works

 sed -n -e '/^[0-9]/s/ *//gp' file | sed 's/red/red /' | sort -u

Look for all lines beginning with a number and remove all spaces
Look for lines with word 'red' and replace with 'red '
sort unique

4 
 cat abc.txt
1,red ,abc
1,red , a b c
2,blue,x
3,black,y
4,brown,xv
4,brown,x v
4,brown,  x  v

cat abc.txt  | sort | uniq -w 1

HTH,
PL

5

You can use the following code using Perl script

 
open (FH,"<file") or die "Can't open";
my $val=1;
my $value;
 while($value=<FH>)
 {
   if($value=~/^$val/)
   {
       $val=$val+1;
       print "$value";
   }
 }
6 
awk -F"," ' !a[$1$2]++ ' file
7 
$ 
$ cat f9
1,red ,abc
1,red , a b c
2,blue,x
3,black,y
4,brown,xv
4,brown,x v
4,brown,  x  v
$ 
$ perl -lne '/([^,]+,[^,]+)/; print if $1 ne $p; $p = $1' f9
1,red ,abc
2,blue,x
3,black,y
4,brown,xv
$ 
$

tyler_durden