bhas
January 31, 2012, 1:21pm
1
Hi,
I am new to scripting.I had one problem infront of me.I tried in many ways with minimal knowledge........Kindly help me.
Description:
I want a shell script where it has to read an input.txt file and need to remove duplicate lines and the result need to kept in output.txt file.
input file example:
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
1974929729,778483889398899,99488495883994,783948859959
8575785999,849498599595050,90505096000660,885699606000
As above my input file looks ,the first 10 digits seperated by delimiter ,repeted 2 times are more.script need to delete such repeated lines keeping any one line.in above example 1974929729 is repeted 2 times.so the second nee to remove.my input file contains more than 25k lines.
Expected output.txt:
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
8575785999,849498599595050,90505096000660,885699606000
Appreciated your help
system
January 31, 2012, 1:32pm
2
Use sort with the -u option to show only unique lines
sort -u <filename> > outputfile.txt
methyl
January 31, 2012, 1:45pm
3
Does not match specification or the data samples in post #1 . (Lines 1 and 3 are not identical. Only the key field is identical).
Try this:
( was before edit )
cat input.txt | tr ',' ' ' | sort -u -k1,1
( after edit)
tr ',' ' ' < input.txt | sort -u -k1,1 | tr ' ' ','
As for your example:
$ cat input.txt
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
1974929729,778483889398899,99488495883994,783948859959
8575785999,849498599595050,90505096000660,885699606000
$ tr ',' ' ' < input.txt | sort -u -k1,1 | tr ' ' ','
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
8575785999,849498599595050,90505096000660,885699606000
awk -F , '!a[$1]++' input.txt
bhas
February 3, 2012, 3:59am
6
Thanks much to all.
frederik,
Special thanks.Your code perfectly suited to my problem..