Compare files in two folders and delete missing ones

Shell Programming and Scripting
1

I do not know much about shell scripting so I am at a loss here. If someone can help me, that would be great!

I have two directories

/dir1

/dir2

I need to delete all files from /dir1 and that does not have a correspondent file in /dir2. It should NOT check file suffixes in /dir2 . Why? Because in /dir2 i have .torrent files and in /dir1 I have the correspondent source file.

ex: (will all present files in both directories)
/dir1/openbsd4_6.iso
/dir1/openbsd4_1.iso
/dir1/freebsd7i386.iso
/dir1/freebsd7amd64.iso

/dir2/openbsd4_6.iso.torrent
/dir2/freebsd7amd64.iso.torrent

In this case /dir1/openbsd4_1.iso and /dir1/freebsd7i386.iso should be removed (from /dir1).

Thanks in advance if you can help.

2

Try something like this:

while read file
do
  if [ ! -e /dir2/${file}.torrent ]
  then
    rm ${file}		# try this first with ls instead of rm
  fi
done < *.iso
3

Replace the echo statement with remove statment below

as

rm $file

#!/bin/sh

for file in dir1/*
do
filestr=`basename $file`
if [ `ls dir2 | grep -c $filestr` -eq 0 ]; then
echo "$file not exists in dir2"
fi
done

cheers,
Devaraj Takhellambam

4

You use the following script to do remove the files from dir1 which files are existing in dir2.

j=0
for i in `ls dir1 `
do
arr[$j]=$i
let j+=1
done
for i in `ls dir2`
do
if [[ -f dir2/$i ]];then
file=`echo $i | cut -d '.' -f 1-2`
for((k=0;k<${#arr[@]};k++))
do
if [[ ${arr[$k]} == $file ]]
then
rm dir1/$file
fi
done
fi
done
5 
for line in `ls -l . | sed "1d" |tr -s ' '|cut -d' ' -f 9`
do
        original_file_name=`echo $line`
        file_name=`echo $line | cut -d'.' -f 1`
#       echo $file_name
        for line in `ls -l ../04/ | sed "1d" |tr -s ' '|cut -d' ' -f 9`
        do
                file_name_in_another_dir=`echo $line | cut -d'.' -f 1`
#               echo $file_name_in_another_dir
                if [ $file_name == $file_name_in_another_dir ]
                then
                        echo "deleting file from ./$original_file_name"
                        rm "$original_file_name" >/dev/null
                fi
        done
done

Try the above bash script. This is also used to delete the files. At first it will compare all the files in the current directory
with the file names under the ../04 directory.

What are the file names are getting matched then it will delete those files from the current directory.

6

Use diff command

diff dir1 dir2

it will give the filenames , you can use any parsing method and remove that

7

Thanks for all your responses!
As I understand it it is not possible to a straight directory to directory comparisson since the file names are not identical (the suffixes and the last part of the filenames are different, .iso.torrent vs .iso).
Ex: /dir2/openbsd4_6.iso.torrent vs /dir1/openbsd4_6.iso

Which solutions take this into account? Also all files are not .iso. Some are .rar etc.

8

To read all the files in the current directory and compare them with the .torrent files in /dir2 you can change *.iso to * in my script:

while read file
do
  if [ ! -e /dir2/${file}.torrent ]
  then
    rm ${file}		# try this first with ls instead of rm
  fi
done < *
9

The basic idea seems to exactly what I am looking for but its not quite working . I tried your example first (didnt work) I then did some minor changes but I was unsuccessful.

#!/bin/sh

while read file in /www/prem/142
do
  if [ ! -e /www/torrent/142/${file}.torrent ]
  then
    ls ${file}                # try this first with ls instead of rm
  fi
done < *.rar

I get this error message:
/home/kaah/torrent_clean.sh: cannot open *.rar: No such file or directory

---------- Post updated at 08:14 AM ---------- Previous update was at 08:10 AM ----------

Thanks for your update. Almost there I think.

#!/bin/sh

while read file
do
  if [ ! -e /www/torrent/142/${file}.torrent ]
  then
    ls ${file}          # try this first with ls instead of rm
  fi
done < *

This give me:
/home/kaah/torrent_clean.sh: cannot open *: No such file or directory

Any ideas? I run this script in the folder where I want to delete files.

10

Oops, sorry for the mistake, should be:

#!/bin/sh

for file in *
do
  if [ ! -e /www/torrent/142/${file}.torrent ]
  then
    ls ${file}          # try this first with ls instead of rm
  fi
done
11

Thanks, no error message now but nothings seems to happen. It just sits there until I abort it.

12

Try to find out what's going wrong, do you get any output with this?

#!/bin/sh

for file in *
do
  ls ${file}          # file current directory
  ls /www/torrent/142/${file}.torrent ]		# files torrent directory
done
13

Doing a test on a test dir:

ks2014631# /home/kaah/torrent_clean2.sh
test.file
ls: ]: No such file or directory
/home/kaah/test1/test.file.torrent
test1.file
ls: ]: No such file or directory
/home/kaah/test1/test1.file.torrent
test2.file
ls: /home/kaah/test1/test2.file.torrent: No such file or directory
ls: ]: No such file or directory
test3.file
ls: ]: No such file or directory
/home/kaah/test1/test3.file.torrent
test4.file
ls: /home/kaah/test1/test4.file.torrent: No such file or directory
ls: ]: No such file or directory

#!/bin/sh


for file in *
do
  ls ${file}          # file current directory
  ls /home/kaah/test1/${file}.torrent ]         # files torrent directory
done
14

There was a typo in the code, try:

#!/bin/sh

for file in *
do
  ls ${file}          # file current directory
  ls /home/kaah/test1/${file}.torrent         # files torrent directory
done
15

Output:

ks2014631# /home/kaah/torrent_clean2.sh
ls: spaces.file: No such file or directory
ls: test: No such file or directory
ls: with: No such file or directory
ls: /home/kaah/test1/test: No such file or directory
ls: spaces.file.torrent: No such file or directory
ls: with: No such file or directory
test.file
/home/kaah/test1/test.file.torrent
test1.file
/home/kaah/test1/test1.file.torrent
test2.file
ls: /home/kaah/test1/test2.file.torrent: No such file or directory
test3.file
/home/kaah/test1/test3.file.torrent
test4.file
ls: /home/kaah/test1/test4.file.torrent: No such file or directory

Dir contents:

ks2014631# ls -l /home/kaah/test1
total 8
-rw-r--r--  1 root  wheel  2 Mar  3 14:19 test with spaces.file.torrent
-rw-r--r--  1 root  wheel  2 Mar  3 13:41 test.file.torrent
-rw-r--r--  1 root  wheel  2 Mar  3 13:41 test1.file.torrent
-rw-r--r--  1 root  wheel  2 Mar  3 13:41 test3.file.torrent
ks2014631# ls -l /home/kaah/test2
total 12
-rw-r--r--  1 root  wheel  2 Mar  3 14:19 test with spaces.file
-rw-r--r--  1 root  wheel  2 Mar  3 13:42 test.file
-rw-r--r--  1 root  wheel  2 Mar  3 13:42 test1.file
-rw-r--r--  1 root  wheel  2 Mar  3 13:42 test2.file
-rw-r--r--  1 root  wheel  2 Mar  3 13:42 test3.file
-rw-r--r--  1 root  wheel  2 Mar  3 13:42 test4.file

Adding " " to the commands make it work with filenames with spaces.
It seems like the if is not working properly. It gets looped even though the file is missing.

16

If you have spaces in the file names you have to quote the variables:

#!/bin/sh

for file in *
do
  ls "${file}"          # file current directory
  ls /home/kaah/test1/"${file}".torrent         # files torrent directory
done
17

Code:

#!/bin/sh

for file in *
do
 if [ ! -e /www/kaah/test1/"${file}".torrent ]
then
   # ls "${file}"          # try this first with ls instead of rm
   # ls "/home/kaah/test1/${file}.torrent"
    ls "${file}"          # file current directory
    ls /home/kaah/test1/"${file}".torrent         # files torrent directory
fi
done

Output:

ks2014631# /home/kaah/torrent_clean2.sh
test with spaces.file
/home/kaah/test1/test with spaces.file.torrent
test.file
/home/kaah/test1/test.file.torrent
test1.file
/home/kaah/test1/test1.file.torrent
test2.file
ls: /home/kaah/test1/test2.file.torrent: No such file or directory
test3.file
/home/kaah/test1/test3.file.torrent
test4.file
ls: /home/kaah/test1/test4.file.torrent: No such file or directory

The if does not seem to work. It loops if though the file is not missing.

18

Hi.

It seems that filenames are getting in the way. Perhaps incorporating the checksums would help. See man md5sum, man shasum ... cheers, drl

19

Can you checksum just the filenames?

20

A typo in your code:

#!/bin/sh

for file in *
do
 if [ ! -e /www/kaah/test1/"${file}".torrent ]
then
   # ls "${file}"          # try this first with ls instead of rm
   # ls "/home/kaah/test1/${file}.torrent"
    ls "${file}"          # file current directory
    ls /www/kaah/test1/"${file}".torrent         # files torrent directory
fi
done