Execution problems with scripting

Hi,
I am new to scripting.I had one problem infront of me.I tried in many ways with minimal knowledge........Kindly help me.
Description:

I want a shell script where it has to read an input.txt file and need to remove duplicate lines and the result need to kept in output.txt file.

input file example:

1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
1974929729,778483889398899,99488495883994,783948859959
8575785999,849498599595050,90505096000660,885699606000

As above my input file looks ,the first 10 digits seperated by delimiter ,repeted 2 times are more.script need to delete such repeated lines keeping any one line.in above example 1974929729 is repeted 2 times.so the second nee to remove.my input file contains more than 25k lines.
Expected output.txt:

1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
8575785999,849498599595050,90505096000660,885699606000
 

Appreciated your help :slight_smile:

Use sort with the -u option to show only unique lines

sort -u <filename> > outputfile.txt

Does not match specification or the data samples in post #1. (Lines 1 and 3 are not identical. Only the key field is identical).

Try this:

( was before edit )

cat input.txt | tr ',' ' ' | sort -u -k1,1

( after edit)

tr ',' ' ' < input.txt | sort -u -k1,1 | tr ' ' ','

As for your example:

$ cat input.txt 
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
1974929729,778483889398899,99488495883994,783948859959
8575785999,849498599595050,90505096000660,885699606000


$ tr ',' ' ' < input.txt | sort -u -k1,1 | tr ' ' ','
1974929729,388004948949994,8849490049004,7894994590599
7240283747,778938934988489,87888948949894,998389499499
8575785999,849498599595050,90505096000660,885699606000
awk -F , '!a[$1]++' input.txt 

Thanks much to all.

frederik,

Special thanks.Your code perfectly suited to my problem..:slight_smile: