Remove Duplicate by considering multiple columns

Shell Programming and Scripting
1

hi friends,

my input

chr1	exon	35204	35266	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	42357	42473	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	45261	45404	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	50701	50778	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	51380	51391	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	51649	51846	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	51961	52077	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	52462	52695	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	53305	53451	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	53497	53778	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	53914	54087	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	54187	54399	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	55691	55996	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	56045	56365	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	56636	56986	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	57161	57304	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	57335	57403	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	59371	59407	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	60822	60878	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	61836	61919	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	63192	63230	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	63393	63425	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	66019	66156	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	35204	35266	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	42357	42473	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	45261	45404	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	50701	50778	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	51380	51391	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	51649	51846	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	51961	52077	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	52462	52695	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	53305	53451	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	53497	53778	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	53914	54087	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	54187	54399	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	55691	55996	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	56045	56365	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	56636	56986	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	57161	57304	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	57335	57403	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	59371	59407	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	60822	60878	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	61836	61919	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	63192	63230	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	63393	63425	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	66019	66156	gene_id "GOLGB1"; transcript_id "GOLGB1_dup1";
chr1	exon	35204	35266	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	42357	42473	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	45261	45404	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	50701	50778	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	51380	51391	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	51649	51846	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	51961	52077	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	52462	52695	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	53305	53451	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	53497	53778	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	53914	54087	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	54187	54399	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	55691	55996	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	56045	56365	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	56636	56986	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	57161	57304	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	57335	57403	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	59371	59407	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	60822	60878	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	61836	61919	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	63192	63230	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	63393	63425	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";
chr1	exon	66019	66156	gene_id "GOLGB1"; transcript_id "GOLGB1_dup2";

my output

chr1	exon	35204	35266	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	42357	42473	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	45261	45404	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	50701	50778	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	51380	51391	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	51649	51846	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	51961	52077	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	52462	52695	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	53305	53451	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	53497	53778	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	53914	54087	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	54187	54399	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	55691	55996	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	56045	56365	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	56636	56986	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	57161	57304	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	57335	57403	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	59371	59407	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	60822	60878	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	61836	61919	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	63192	63230	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	63393	63425	gene_id "GOLGB1"; transcript_id "GOLGB1";
chr1	exon	66019	66156	gene_id "GOLGB1"; transcript_id "GOLGB1";

I have to compare on the first 4 columns for this file. The delimiters are both tab and space. I would really appreciate anything in awk that would make me flexible to add or delete the no. of columns to consider for matching. That would make my task easier in the future, if I have to work on other files with more no. of columns.

Thanks in advance.

2 
awk '!a[$1$2$3$4]++' filename
1 Like
3

I tried

.

I lost my brain to think in a more simplified fashion :D:D:D

Gracias my friend!