# remove duplicate files in a directory

**URL:** <https://community.unix.com/t/remove-duplicate-files-in-a-directory/160208>\
**Category:** Shell Programming and Scripting\
**Created:** [March 13, 2006, 2:32pm UTC](https://community.unix.com/t/remove-duplicate-files-in-a-directory/160208 "2006-03-13T14:32:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![asinha63](https://community.unix.com/letter_avatar/asinha63/32/5_5575768a8748004e209b776fc1b2916d.png) [@asinha63](https://community.unix.com/u/asinha63)\
**Post date:** [March 13, 2006, 2:32pm UTC](https://community.unix.com/t/remove-duplicate-files-in-a-directory/160208/1 "2006-03-13T14:32:22Z")

</div>

Hi ppl.  
I have to check for duplicate files in a directory .  
the directory has following files  
/the/folder /containing/the/file  
a1.yyyymmddhhmmss  
a1.yyyyMMddhhmmss  
b1.yyyymmddhhmmss  
b2.yyyymmddhhmmss  
c.yyyymmddhhmmss  
d.yyyymmddhhmmss  
d.yyyymmddhhmmss

where the date time stamp can be different for the same file hence the risk of duplicate files.  
How do i make the validate so that there are no duplicate files to be processed .  
Anubha

---

<div class="post-metadata">

**Author:** ![jim\_mcnamara](https://community.unix.com/letter_avatar/jim_mcnamara/32/5_5575768a8748004e209b776fc1b2916d.png) [@jim\_mcnamara](https://community.unix.com/u/jim_mcnamara)\
**Post date:** [March 13, 2006, 2:46pm UTC](https://community.unix.com/t/remove-duplicate-files-in-a-directory/160208/2 "2006-03-13T14:46:21Z")

</div>

PS: one directory doesn't have duplicated names...

if you run cksum or another hash like md5 you can find duplicates that way:

```nohighlight
cd /path/to/wherever
for file in `ls *`
do
      cksum $file
done | awk '{ 
         arr[$1]++
         if(arr[$1]>1) {print $0 }
         } ' > ./duplicate.files

```