How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format.
aaa|1234|2010-12-31
aaa|1234|2010-11-10
bbb|345|2011-01-01
ccc|346|2011-02-01
bbb|345|2011-03-10
aaa|1234|2010-01-01
Required Output
aaa|1234|2010-12-31
bbb|345|2011-03-10
ccc|346|2011-02-01
I tried using sort -u , but how to retain the record with maximum date.
Thanks
Arif
awk -F'[|-]' '{a[$1"|"$2]=($3+$4+$5)>t?$3"-"$4"-"$5:a[$1"|"$2];t=$3+$4+$5}END{for(i in a) print i,a|"sort"}'
Thanks
IF the file is already sorted on the three columns , can the remove duplicate part be a little simpler , I wanted to use this code in a datastage program , and was wondering if the code can be little simpler so that other datstage developers ( less unix background) can understand .
how about this?
echo 'aaa|1234|2010-12-31
aaa|1234|2010-11-10
bbb|345|2011-01-01
ccc|346|2011-02-01
bbb|345|2011-03-10
aaa|1234|2010-01-01' |sort -t '|' -k1,1 -k3,3r |awk -F"|" '++a[$1"|"$2]==1'
aaa|1234|2010-12-31
bbb|345|2011-03-10
ccc|346|2011-02-01
yinyuemi:
how about this?
echo 'aaa|1234|2010-12-31
aaa|1234|2010-11-10
bbb|345|2011-01-01
ccc|346|2011-02-01
bbb|345|2011-03-10
aaa|1234|2010-01-01' |sort -t '|' -k1,1 -k3,3r |awk -F"|" '++a[$1"|"$2]==1'
aaa|1234|2010-12-31
bbb|345|2011-03-10
ccc|346|2011-02-01
sort -t '|' -k3,3r infile |awk -F \| '!a[$1 FS $2]++'