Delete duplicates via script?

Shell Programming and Scripting
1

Hello,
i have the following problem:
there are two folders with a lot of files.
Example:
FolderA contains AAA, BBB, CCC
FolderB contains DDD, EEE, AAA

How can i via script identify AAA as duplicate in Folder B and delete it there? So that only DDD and EEE remain, in Folder B?

Thank you in advance for looking into this.

2

No two files with same name can be in the same folder.

3

that is why they are in Folder A and Folder B, as i wrote :wink:

4

which env? Is the diff command in your system support to diff folder?

5

when i enter "diff" at the command line, it is recognized as command. I am using ubuntu, if this is any useful information :wink:

The aim is to keep the files each in their folders, but just delete in B those which already exist in A

6 
#!/bin/sh
vr=$(ls /path/to/FolderA)
for fl1 in ${vr};do
    echo "${fl1}"
done

vr2=$(ls /path/to/FolderB)
for fl2 in ${vr2};do
    echo "${fl2}"
done

Now make two arrays to save the result and compare those two arrays.
If there is any match then remove those files.

1 Like
7

in ubuntu, you should be able to diff the folder directly.

diff /path/to/FolderA /path/to/FolderB
1 Like
8

ok, thank you, i will try the Diff.. the array option would mean still i have to compare manually if i read it right, which is futile because it is >10K files

9 
#!/usr/bin/env ruby  -w

require 'fileutils'
Dir["./A/*"].each do |d|
  name=File.basename(d) if File.directory?(d)
  if File.exists?("B/"+name)
    print "#{name} exists in #{"B/"+name}\n"
    print "Deleting %s\n" % name
    FileUtils.rm_r(d, :force => true)
  end
end
1 Like
10

Ah, that works too, kurumi, thank you :wink:

11 
for i in FolderA;do
  ls ${i}/* | while read file;do
    tfile=`basename $file`
    if [ -f FolderB/${tfile} ];then
      rm FolderB/${tfile}
    fi
  done
done
1 Like
12

Simple
while read file
do rm -f FolderB/$file
done < <(ls FolderA)

1 Like
13

Note: that is bash code, not shell code.

14

According to Wikipedia :smiley:

I think any (command line style) scripting code that is designed to work in shell is "shell code" isn't it?

15

OK, maybe I should have said: That is bash shell code, not POSIX compliant shell code, so it will only work in bash or maybe ksh93. :slight_smile:

---------- Post updated at 10:22 ---------- Previous update was at 10:12 ----------

for i in FolderA/*
do
  rm -f "FolderB${i#FolderA}" 2>/dev/null
done

ls FolderA |
while read file
do 
  rm -f "FolderB/$file" 2>/dev/null
done
2 Likes
16

There's a also slight difference in the meaning of "shell code" in computer security. In computer security, a shellcode is a small piece of code used as the payload in the exploitation of a software vulnerability (as defined by Wikipedia). Its not to be confused with "shell scripting code". :slight_smile:

1 Like
17

you all have been of great help. Thank you so very much!