# Select distinct values from a flat file

**URL:** <https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959>\
**Category:** Shell Programming and Scripting\
**Created:** [May 18, 2010, 2:04pm UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959 "2010-05-18T14:04:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![smalya](https://community.unix.com/letter_avatar/smalya/32/5_5575768a8748004e209b776fc1b2916d.png) [@smalya](https://community.unix.com/u/smalya)\
**Post date:** [May 18, 2010, 2:04pm UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/1 "2010-05-18T14:04:18Z")

</div>

Hi ,  
I have a similar problem.  
Please can anyone help me with a shell script or a perl.  
I have a flat file like this

```nohighlight
fruit country
apple germany
apple india
banana pakistan
banana saudi
mango india

```

I want to get a output like

```nohighlight
 
fruit country
apple germany
banana pakistan
mango india

```

Is there anyways this can be done?:mad:😕

---

<div class="post-metadata">

**Author:** ![danmero](https://community.unix.com/letter_avatar/danmero/32/5_5575768a8748004e209b776fc1b2916d.png) [@danmero](https://community.unix.com/u/danmero)\
**Post date:** [May 18, 2010, 2:11pm UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/2 "2010-05-18T14:11:10Z")

</div>

```nohighlight
awk '{a[$1]++}a[$1]<2' file

```

---

<div class="post-metadata">

**Author:** ![smalya](https://community.unix.com/letter_avatar/smalya/32/5_5575768a8748004e209b776fc1b2916d.png) [@smalya](https://community.unix.com/u/smalya)\
**Post date:** [May 18, 2010, 2:14pm UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/3 "2010-05-18T14:14:42Z")

</div>

Thanks sir. Please also tell me how this can be sent to a file

---

<div class="post-metadata">

**Author:** ![danmero](https://community.unix.com/letter_avatar/danmero/32/5_5575768a8748004e209b776fc1b2916d.png) [@danmero](https://community.unix.com/u/danmero)\
**Post date:** [May 18, 2010, 4:09pm UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/4 "2010-05-18T16:09:19Z")

</div>

```nohighlight
awk '{a[$1]++}a[$1]<2' file > new.file

```

---

<div class="post-metadata">

**Author:** ![smalya](https://community.unix.com/letter_avatar/smalya/32/5_5575768a8748004e209b776fc1b2916d.png) [@smalya](https://community.unix.com/u/smalya)\
**Post date:** [May 19, 2010, 2:37am UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/5 "2010-05-19T02:37:01Z")

</div>

Thanks Mate for this.  
Unforfunately other module have decided to send a comma seperated file instead of space seperated. Please help again .  
I have values like this now

```nohighlight
INDIA,mango,2
PAK,mango,1
USA,apple,1
UK,apple,1

```

---

<div class="post-metadata">

**Author:** ![malcomex999](https://community.unix.com/letter_avatar/malcomex999/32/5_5575768a8748004e209b776fc1b2916d.png) [@malcomex999](https://community.unix.com/u/malcomex999)\
**Post date:** [May 19, 2010, 2:40am UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/6 "2010-05-19T02:40:32Z")

</div>

> [@smalya](#):
>
> Thanks Mate for this.  
> Unforfunately other module have decided to send a comma seperated file instead of space seperated. Please help again .  
> I have values like this now
> 
> INDIA,mango,2  
> PAK,mango,1  
> USA,apple,1  
> UK,apple,1

A little modificatiion to danmero solution...

```plaintext
awk -F, '{a[$1]++}a[$1]<2' file > new.file

```

i guess you might need to change $1 to $2...

---

<div class="post-metadata">

**Author:** ![devtakh](https://community.unix.com/letter_avatar/devtakh/32/5_5575768a8748004e209b776fc1b2916d.png) [@devtakh](https://community.unix.com/u/devtakh)\
**Post date:** [May 19, 2010, 2:41am UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/7 "2010-05-19T02:41:50Z")

</div>

```nohighlight
awk -F "," '{a[$2]++}a[$2]<2' file2

```

cheers,  
Devaraj Takhellambam

---

<div class="post-metadata">

**Author:** ![Franklin52](https://community.unix.com/user_avatar/community.unix.com/franklin52/32/500_2.png) [@Franklin52](https://community.unix.com/u/Franklin52)\
**Post date:** [May 19, 2010, 5:24am UTC](https://community.unix.com/t/select-distinct-values-from-a-flat-file/265959/8 "2010-05-19T05:24:06Z")

</div>

Shorter:

```nohighlight
awk -F "," '!a[$2]++' file

```