Remove duplicate files

Shell Programming and Scripting
1

Hi,

In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named:

 
12345___PP___0902___AA.txt
12346___PP___0902___AA. txt
12347___PP___0902___AA. txt

The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove others in a new dir. Could you help me? Is there any script?
Thank you for your helping in advance.

2

Off the top of my head, test before using in the real world!!

latest=0;
for i in $(find ~/corpus -name \*___PP___0902___AA. txt); do
    current=$(echo $(basename $1) \ cut -d_ -f1) 
    if [ $current -lt $latest ] ; then
        #rm $i
        echo we would delete $i" # I HAVE NOT TESTED THIS CODE
    else
        latest=$current
    fi
done
3

thanks for the reply! Is that a bash? Sorry I am new in linux.
Can I replace *___PP___0902___AA. txt with *.txt because there are file names named

12345___PP___0902___AA.txt
12346___PP___0902___AA. txt
12347___PP___0902___AA. txt
12345___PP___0903___AA.txt
12346___PP___0903___AA. txt
12347___PP___0903___AA. txt
...

and another question. If I need the file with the smallest ID?

Thanks a lot!

4

Yes, it's bash.
You could certainly use \*.txt in the find command
If you need to build a list of not the highest or lowest you could use an array or a temp file and set the first to be both highest and lowest
The rm command should be replaced writing the names of the files to be moved to your array/temp file
Then post process with a mv command for everything in the array / temp file , possibly recreating any directory structure underneath with a mkdir -p $new_dir/$(basenaname $i)

5

I am sorry but it does not work. I can not run it.

dupl.sh: line 8: unexpected EOF while looking for matching `"'
dupl.sh: line 13: syntax error: unexpected end of file
6 
latest=0;
back_up="$HOME/backup" # or whatever makes sense for your use case
 for i in $(find ~/corpus -name \*.txt); do     current=$(echo $(basename $1) \ cut -d_ -f1)      if [ $current -lt $latest ] ; then
        if [ ! -d $back_up/$(dirname $i) ] ; then
            #mkdir -p $back_up/$(dirname $i)
            echo "We would have created the $back_up/$(dirname $i) directory"         fi
        # mv $i $back_up/$(dirname $i)         echo "We would have moved $i to $back_up/$(dirname $i)" # I HAVE NOT TESTED THIS CODE     else         latest=$current     fi done

Strangely enough, untested stream of conciousness code doesn't work, fortunately the parser explained why it baulked, there was a terminating quote on a string with no opening quote.
Try the above and then modify as outlined