# Get a none duplicate list file

**URL:** <https://community.unix.com/t/get-a-none-duplicate-list-file/142998>\
**Category:** Shell Programming and Scripting\
**Created:** [June 14, 2002, 10:54am UTC](https://community.unix.com/t/get-a-none-duplicate-list-file/142998 "2002-06-14T10:54:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![trynew](https://community.unix.com/letter_avatar/trynew/32/6_22734026db12803ebb3e059622ac3534.png) [@trynew](https://community.unix.com/u/trynew)\
**Post date:** [June 14, 2002, 10:54am UTC](https://community.unix.com/t/get-a-none-duplicate-list-file/142998/1 "2002-06-14T10:54:28Z")

</div>

Dear sir

```
     i got a file like following format with the duplicate line:

       AAA 
       AAA 
       AAA 
       AAA 
       AAA 
       BBB
       BBB
       BBB
       BBB
       CCC
       CCC
       CCC
       CCC

   can help me to shorten the file size by one record apear only once in the file? Thanks in advance!!!
```

---

<div class="post-metadata">

**Author:** ![usfrog](https://community.unix.com/letter_avatar/usfrog/32/6_22734026db12803ebb3e059622ac3534.png) [@usfrog](https://community.unix.com/u/usfrog)\
**Post date:** [June 14, 2002, 11:24am UTC](https://community.unix.com/t/get-a-none-duplicate-list-file/142998/2 "2002-06-14T11:24:05Z")

</div>

sort will do (look at the man page for more options):

cat myFile | sort -u \> myFileNoDup # create 2nd file  
mv myFileNoDup myFile # replace first file

---

<div class="post-metadata">

**Author:** ![trynew](https://community.unix.com/letter_avatar/trynew/32/6_22734026db12803ebb3e059622ac3534.png) [@trynew](https://community.unix.com/u/trynew)\
**Post date:** [June 15, 2002, 12:23pm UTC](https://community.unix.com/t/get-a-none-duplicate-list-file/142998/3 "2002-06-15T12:23:26Z")

</div>

Thank you very much!!!

---

<div class="post-metadata">

**Author:** ![peter.herlihy](https://community.unix.com/letter_avatar/peter.herlihy/32/6_22734026db12803ebb3e059622ac3534.png) [@peter.herlihy](https://community.unix.com/u/peter.herlihy)\
**Post date:** [June 20, 2002, 1:47am UTC](https://community.unix.com/t/get-a-none-duplicate-list-file/142998/4 "2002-06-20T01:47:42Z")

</div>

You can just use the syntax

sort -u myfile -o myfile

Cat is redundant in the suggested statement above and the sort command allows output to the input filename (unlike sed etc.)

Do you wish to retain the order in the file....and just remove duplicates? Or do you want to sort and remove duplicates?

sort -u -m myfile -o myfile

Will remove only duplicates that appear directly next door. It assumes the list is already sorted and looks for two rows next to each other that are the same. So a list of

B  
B  
A  
A  
B  
B  
A  
A

Will come back as

B  
A  
B  
A

------------

---

<div class="post-metadata">

**Author:** ![yeheyaansari](https://community.unix.com/letter_avatar/yeheyaansari/32/6_22734026db12803ebb3e059622ac3534.png) [@yeheyaansari](https://community.unix.com/u/yeheyaansari)\
**Post date:** [June 20, 2002, 9:57am UTC](https://community.unix.com/t/get-a-none-duplicate-list-file/142998/5 "2002-06-20T09:57:38Z")

</div>

You can use the uniq command for the same

uniq filename

Thanks  
Yeheya

---

<div class="post-metadata">

**Author:** ![Nisha](https://community.unix.com/letter_avatar/nisha/32/6_22734026db12803ebb3e059622ac3534.png) [@Nisha](https://community.unix.com/u/Nisha)\
**Post date:** [June 25, 2002, 12:06am UTC](https://community.unix.com/t/get-a-none-duplicate-list-file/142998/6 "2002-06-25T00:06:49Z")

</div>

Hi,

The -u option in sort also stands for unique. It suppresses all the duplicate keys except one. There is another option -c which checks whether the single input file is sorted.

-Nisha