Deleting lines containing duplicated strings

Shell Programming and Scripting
1

Dear all,

I always appreciate your help.

I would like to delete lines containing duplicated strings in the second column.

test.txt

658	invert_d2e_q_reg_0_/Qalu_ecl_zlow_e	0.825692
659	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[31]	0.825692
660	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[63]	0.825692
661	invert_d2e_q_reg_0_/Qalu_ecl_zhigh_e	0.825692
665	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[62]	0.825692
666	invert_d2e_q_reg_0_/Qalu_ecl_zlow_e	0.825692
668	invert_d2e_q_reg_0_/Qalu_ecl_zhigh_e	0.825692
670	invert_d2e_q_reg_0_/Qalu_ecl_zhigh_e	0.825692
673	invert_d2e_q_reg_0_/Qalu_ecl_zlow_e	0.825692
675	invert_d2e_q_reg_0_/Qalu_ecl_zhigh_e	0.825692
677	invert_d2e_q_reg_0_/Qalu_ecl_zhigh_e	0.825692
678	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[27]	0.825692
679	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[27]	0.8120
.
.
.

output.txt

658	invert_d2e_q_reg_0_/Qalu_ecl_zlow_e	0.825692
659	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[31]	0.825692
660	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[63]	0.825692
661	invert_d2e_q_reg_0_/Qalu_ecl_zhigh_e	0.825692
665	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[62]	0.825692
678	invert_d2e_q_reg_0_/Qalu_byp_rd_data_e[27]	0.825692
.
.
.

I know sed can delete lines with predefined specific strings, but in my cases, I could not expect the strings are duplicated. Also, duplicated strings will be more than 1000.

I used �uniq� to do this job, but this does not work.
uniq -u -f 4 test.txt
(-u prints unique lines. -f skips the first 4 letters. )

Is there any way to do this with sed/awk/perl? Or please correct my uniq semantics.

Best,

Jaeyoung

2

Try:

awk '!H[$2]++' infile

Or using sort you could sort on 2nd field and print 1st of an equal run, then re-sort back into numeric order on field 1:

sort -u -k 2,2 infile | sort -n
1 Like