Script to tar/rsync/rm multiple folder names

Shell Programming and Scripting
41

Hi.

I see that there are similar recent questions posted in
wait and continue if directory stays same size
and in
wait and continue if directory stays same size

My take is that directory space is allocated not just one entry at a time, but by blocks. So that there may be no noticeable change when objects are added to a directory.

Similarly, on some systems (perhaps older), directory space was not reduced when object are deleted.

A search provides these links, describing such properties as directory size:
linux - Shrink/reset directory size? - Server Fault
inode - Monotonic growth of Linux directory size/block count - Server Fault
https://bbs.archlinux.org/viewtopic.php?id=135252

So trying to detect a change in directory content by monitoring the size seems to be fruitless.

However, there are, as can be seen from the responses at LQ, that there may be ways to monitor events within a directory. (I thought I had posted a demonstration of that, but I cannot find it just now.)

Best wishes ... cheers, drl

---------- Post updated at 06:17 ---------- Previous update was at 06:08 ----------

Hi.

Exactly.

In my situation, I almost never format/indent manually because I'm too fallible and there are tools to do this. I apply these continually while I am developing. Here are some:

Tidy, format, indent, beautify, pretty-print source code text files

        0) pretty-printing on postscript printer
           trueprint
           a2ps

        1) perltidy

        2) awk-pretty
           ftp://ftp.armory.com/pub/scripts/fmtawksh (verified 2016.04.25)
           http://www.armory.com/~ftp/

        3) f90tidy
           http://www.ifremer.fr/ditigo/molagnon/fortran90/ (verified 2016.04.25)

        4) f77tidy
           http://www.pdas.com/tidy.html (verified 2016.04.25)

        5) bash, shell
           http://arachnoid.com/python/beautify_bash_program.html
           (verified 2016.04.25)

        6) tclfrink
           http://wiki.tcl.tk/2611 (verified 2016.04.25)
           tcltidy
           http://wiki.tcl.tk/15731 (verified 2016.04.25)

        7) ruby
           http://arachnoid.com/ruby/rbeautify.rb.txt (verified 2016.04.25)
           ruby-beautify

        8) C, C++, Objective-C, C#, and Java
           astyle
           C
           indent

Best wishes ... cheers, drl

42

dont believe it, on the last hurdle i get this error while running my script

bash: /archive_script.sh: /bin/bash^M: bad interpreter: No such file or directory

this is my code

#!/bin/bash
cd /to_be_archived/
for DIR in * ;

        do
        MailAddress="robertw@molinare.co.uk"

        if ! inotifywait -rq -e modify,create,delete -t 60 /to_be_archived/"$DIR" ; then
        mail -s "Modified $DIR" $MailAddress <<< "$DIR failed has been modified within the last minute"
        continue
        fi

        if ! tar cf "$DIR".tar "$DIR" ; then
        mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
        rm -f "$DIR".tar
        continue
        fi

        if [ -f /archived_projects/"$DIR".tar ]; then
        mail -s "duplicate exists $DIR" $MailAddress <<< "$DIR.tar already exists"
        rm -f "$DIR".tar
        continue
        fi

        if ! rsync -a "$DIR".tar /archived_projects/ ; then
        mail -s "rsync failed $DIR" $MailAddress <<< "rsync of $DIR failed due to error, removing $DIR.tar"
        rm -f "$DIR".tar
        continue
        fi

        if ! rm -f "$DIR".tar ; then
        mail -s "remove tar failed $DIR" $MailAddress <<< "removing of tar $DIR failed due to error"
        continue
        fi

        if ! rm -rf "$DIR" ; then
        mail -s "remove folder failed $DIR" $MailAddress <<< "removing of $DIR failed due to error"
        continue
                else
                mail -s "success $DIR" $MailAddress <<< "successfully completed archiving $DIR"
        fi

        done
43

Hi.

Did you edit the file in WIndows? ... cheers, drl

1 Like
44

oops, my bad, its because i copied it from another computer, i just had to create new script and chmod it, job done

---------- Post updated 04-27-16 at 04:52 AM ---------- Previous update was 04-26-16 at 06:34 AM ----------

#!/bin/bash
cd /to_be_archived/
for DIR in * ;

        do
        MailAddress="robertw@molinare.co.uk"

        if inotifywait -rq -e modify,create,delete -t 60 "$DIR"/ ; then
                mail -s "Modified $DIR" $MailAddress <<< "$DIR failed has been modified within the last minute"
        continue
        fi

        if ! tar -cf "$DIR".tar "$DIR" ; then
                mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
                rm -f "$DIR".tar
        continue
        fi

        if [ -f /archived_projects/"$DIR".tar ]; then
                mail -s "duplicate exists $DIR" $MailAddress <<< "$DIR.tar already exists"
                rm -f "$DIR".tar
        continue
        fi

        if ! rsync -a "$DIR".tar /archived_projects/ ; then
                mail -s "rsync failed $DIR" $MailAddress <<< "rsync of $DIR failed due to error, removing $DIR.tar"
                rm -f "$DIR".tar
        continue
        fi

        if ! rm -f "$DIR".tar ; then
                mail -s "remove tar failed $DIR" $MailAddress <<< "removing of tar $DIR failed due to error"
        continue
        fi
        
cd "$DIR"
date >> /archive_details.txt
echo -n  "Folder to be archived = " >> /archive_details.txt
pwd >> /archive_details.txt
echo -n  "Number of files =       " >> /archive_details.txt
find . -type f | wc -l >>  /archive_details.txt
echo -n  "Size in GB =            " >> /archive_details.txt
du . -s -B 1G >>  /archive_details.txt
echo " " >>  /archive_details.txt
cd ..

        if ! rm -rf "$DIR" ; then
                mail -s "remove folder failed $DIR" $MailAddress <<< "removing of $DIR failed due to error"
        continue
                else
                mail -s "success $DIR" $MailAddress <<< "successfully completed archiving $DIR"
        fi

        done
45

really struggling with my last command in this script im trying to do

i want to capture the disk usage of the DIR in "to_be_archived" folder and a sleep command for 10 minutes and then run the disk usage command again to see if the two values change

if they have stop doing this DIR and do the next DIR in the "to_be_archived" folder

if the same values match, it can continue doing the tar/rsync/rm commands

im kind of thinking like this -

du --time "$DIR"/
sleep 600
du --time "$DIR"/

but i dont know how get it to do if the two values change exit, if they stay the same continue

the reason i have used the du --time command is it tells me the modification time and i would rather go by this than the folder size, if that makes sense

thanks

46

Could you please explain why you have chosen to ignore my suggestion in post #40 in this thread to do what you are suggesting above?

What you are suggesting above is not only MUCH slower, but also error prone.

  • Why add 10 minutes of processing time to each directory you want to archive?
  • How do you know that a network delay or system scheduling won't stop whatever is copying files into a /to_be_archived/project directory for 10 minutes?
  • How will you know that something didn't accidentally kill the script copying files into a /to_be_archived/project directory before it completed successfully?
  • How will you know that the script copying files into a /to_be_archived/project directory didn't terminate because if was unable to copy a file (or perform some other action) before it completed?
  • How will you know that a bug in the script copying files into a /to_be_archived/project directory didn't cause the script to die before it completed its job?

All of these problems are avoided if you use the approach I suggested in post #40 in this thread. With the approach I suggested in post #40:

  • Scripts copying files into the /to_be_archived file hierarchy can operate independently from the script in this thread.
  • The script being discussed in this thread won't see a directory for a project to the archived until all files for that project have been successfully transferred into the corresponding /to_be_archived/project directory.
  • There is no need to slow down the script being discussed in this thread to try to determine if the directory for a project to be archived is ready to archive.

However, no matter what you do, you still need to verify that $DIR expands to an existing directory (as I mentioned in posts #16 and #24 in this thread).

And, I still question why you don't move the assignment command:

        MailAddress="robertw@molinare.co.uk"

before your loop instead of resetting its value for each value of DIR you process.

47

Hi Don

i dont really understand

so the operators, once they complete their projects, will move their project folders in the dir "to_be_archived" and once in there the script will take over/handle it and dump the output tar to dir "archived_projects"

lets say the operators have just started moving the project folder into the dir "to_be_archived" and my script is running and it intercepts the new project folder but the project folder is still growing in size, i dont want it to start of the process as they are still moving the project folder to the dir "to_be_archived"

thats why i want to put a wait command, to say if its stopped growing the project for 10 minutes it can start off the script but if its still growing, move on to the newxt project folder in the dir "to_be_archived"

---------- Post updated at 11:22 AM ---------- Previous update was at 07:05 AM ----------

what you think?

#!/bin/bash
cd /vol/cha-work/to_be_archived/audio/auto_archive/
for dir in */

        do DIR=$(basename "$dir")

        MailAddress="robertw@molinare.co.uk"

        if find "$DIR" -mmin -10 | grep '.' ; then
                mail -s "Modified $DIR" $MailAddress <<< "$DIR failed has been modified within the last 10 minutes"
        continue
        fi

        if ! tar -cf "$DIR".tar "$DIR" ; then
                mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
                rm -f "$DIR".tar
        continue
        fi

        if [ -f /media/ARCHIVE/archive_now/audio/"$DIR".tar ]; then
                mail -s "duplicate exists $DIR" $MailAddress <<< "$DIR.tar already exists"
                rm -f "$DIR".tar
        continue
        fi

        if ! rsync -a "$DIR".tar /media/ARCHIVE/archive_now/audio/ ; then
                mail -s "rsync failed $DIR" $MailAddress <<< "rsync of $DIR failed due to error, removing $DIR.tar"
                rm -f "$DIR".tar
        continue
        fi

        if ! rm -f "$DIR".tar ; then
                mail -s "remove tar failed $DIR" $MailAddress <<< "removing of tar $DIR failed due to error"
        continue
        fi

cd "$DIR"
date >> /vol/cha-it/it/archive_details.txt
echo -n  "Folder to be archived = " >> /vol/cha-it/it/archive_details.txt
pwd >> /vol/cha-it/it/archive_details.txt
echo -n  "Number of files =       " >> /vol/cha-it/it/archive_details.txt
find . -type f | wc -l >>  /vol/cha-it/it/archive_details.txt
echo -n  "Size in GB =            " >> /vol/cha-it/it/archive_details.txt
du . -s -B 1G >>  /vol/cha-it/it/archive_details.txt
echo " " >>  /vol/cha-it/it/archive_details.txt
cd ..

        if ! rm -rf "$DIR" ; then
                mail -s "remove folder failed $DIR" $MailAddress <<< "removing of $DIR failed due to error"
        continue
                else
                mail -s "success $DIR" $MailAddress <<< "successfully completed archiving $DIR"
        fi

        done
48

Hi Robert,
You didn't pay close enough attention to what I suggested in post #40: I said it would be much better if you would create a hidden directory under /to_be_archived (such as /to_be_archived/.in_progress ), have your operators move everything into a sub-directory under that directory (such as /to_be_archived/.in_progress/project ), and then after that project directory contains everything your operators want to archive, have them issue the command:

mv /to_be_archived/.in_progress/project /to_be_archived/project

Your existing script WILL NOT SEE /to_be_archived/.in_progress nor any directories under it. (That is why it is called a hidden directory.) After your operators have completed copying files into /to_be_archived/.in_progress/project , then, and only then, as the final step they issue the mv command that will make the completed project directory visible to your existing script.

If you don't believe that this works, run the following simple script to see that hidden directories will not be visible to your script:

#!/bin/ksh
# Create a file hierarchy with one hidden project and one visible project:
mkdir junk junk/.hidden junk/.hidden/project2 junk/project1
cd junk
touch project0.tar project1/file1 project1/file2
touch .hidden/project2/hfile1 .hidden/project2/hfile2

echo 'File hierarchy with project2 directory hidden:'
ls -laR	# Note that if the -a is left out, ls will not see hidden files either.
echo

# The following is similar to the code you are using, but skips over artifacts
# left over from failed attempts to archive arvhive0.
for DIR in *
do	if [ ! -d "$DIR" ]
	then	printf 'Skipping non-directory file "%s".\n' "$DIR"
		continue
	fi
	printf 'Found file "%s" before moving hidden directory.\n' "$DIR"
	# Perform the rest of your script to archive this project...
done
printf '\nNote that .hidden and project2 should not appear above.\n\n'

# Add some more files to project2 (This could be done in parallel with running
# your current script since these files are still hidden from your script.
touch .hidden/project2/hfile3 .hidden/project2/hfile4

# Now make the hidden project2 visible 
mv .hidden/project2 project2
echo 'File hierarchy with project2 directory moved out from under hidden directory.'
ls -laR
echo
for DIR in *
do	if [ ! -d "$DIR" ]
	then	printf 'Skipping non-directory file "%s".\n' "$DIR"
		continue
	fi
	printf 'Found file "%s" after moving hidden sub-directory.\n' "$DIR"
done
printf '\nNote that project2 should now appear above but .hidden is still not seen.\n'

# Now clean up after ourselves...
cd -
rm -rf junk

which produces output similar to:

File hierarchy with project2 directory hidden:
total 0
drwxr-xr-x  5 dwc  staff  170 Apr 29 11:18 .
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 ..
drwxr-xr-x  3 dwc  staff  102 Apr 29 11:18 .hidden
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 project0.tar
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 project1

./.hidden:
total 0
drwxr-xr-x  3 dwc  staff  102 Apr 29 11:18 .
drwxr-xr-x  5 dwc  staff  170 Apr 29 11:18 ..
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 project2

./.hidden/project2:
total 0
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 .
drwxr-xr-x  3 dwc  staff  102 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile2

./project1:
total 0
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 .
drwxr-xr-x  5 dwc  staff  170 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file2

Skipping non-directory file "project0.tar".
Found file "project1" before moving hidden directory.

Note that .hidden and project2 should not appear above.

File hierarchy with project2 directory moved out from under hidden directory.
total 0
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 .
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 ..
drwxr-xr-x  2 dwc  staff   68 Apr 29 11:18 .hidden
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 project0.tar
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 project1
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 project2

./.hidden:
total 0
drwxr-xr-x  2 dwc  staff   68 Apr 29 11:18 .
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 ..

./project1:
total 0
drwxr-xr-x  4 dwc  staff  136 Apr 29 11:18 .
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 file2

./project2:
total 0
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 .
drwxr-xr-x  6 dwc  staff  204 Apr 29 11:18 ..
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile1
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile2
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile3
-rw-r--r--  1 dwc  staff    0 Apr 29 11:18 hfile4

Skipping non-directory file "project0.tar".
Found file "project1" after moving hidden sub-directory.
Found file "project2" after moving hidden sub-directory.

Note that project2 should now appear above but .hidden is still not seen.
/Users/dwc/test/unix.com/shell/Script_to_tar,_rsync,_rm_multiple_folder_names
1 Like
49

i see where your coming from Don and yes this would be easier but our operators, dont really know anything about command line terminology and there like old dogs (cant teach an old dog new tricks)

im doing this to make my life alot easier for myself ie so all they have to do is move thier project folders "to_be_archived" and once in there it will wait 10 minutes and if that project folder hasnt been modified in 10 minutes it will then tar/rsync and remove it

this is my final draft

#!/bin/bash
cd /vol/cha-work/to_be_archived/audio/auto_archive/
for dir in */

        do DIR=$(basename "$dir")

        MailAddress="robertw@molinare.co.uk"

        if find "$DIR" -mmin -10 | grep '.' ; then
                mail -s "Modified $DIR" $MailAddress <<< "$DIR failed has been modified within the last 10 minutes"
        continue
        fi

        if ! tar -cf "$DIR".tar "$DIR" ; then
                mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
                rm -f "$DIR".tar
        continue
        fi

        if [ -f /media/ARCHIVE/archive_now/audio/"$DIR".tar ]; then
                mail -s "duplicate exists $DIR" $MailAddress <<< "$DIR.tar already exists"
                rm -f "$DIR".tar
        continue
        fi

        if ! rsync -a "$DIR".tar /media/ARCHIVE/archive_now/audio/ ; then
                mail -s "rsync failed $DIR" $MailAddress <<< "rsync of $DIR failed due to error, removing $DIR.tar"
                rm -f "$DIR".tar
        continue
        fi

        if ! rm -f "$DIR".tar ; then
                mail -s "remove tar failed $DIR" $MailAddress <<< "removing of tar $DIR failed due to error"
        continue
        fi

cd "$DIR"
date >> /vol/cha-it/it/archive_details.txt
echo -n  "Folder to be archived = " >> /vol/cha-it/it/archive_details.txt
pwd >> /vol/cha-it/it/archive_details.txt
echo -n  "Number of files =       " >> /vol/cha-it/it/archive_details.txt
find . -type f | wc -l >>  /vol/cha-it/it/archive_details.txt
echo -n  "Size in GB =            " >> /vol/cha-it/it/archive_details.txt
du . -s -B 1G >>  /vol/cha-it/it/archive_details.txt
echo " " >>  /vol/cha-it/it/archive_details.txt
cd ..

        if ! rm -rf "$DIR" ; then
                mail -s "remove folder failed $DIR" $MailAddress <<< "removing of $DIR failed due to error"
        continue
                else
                mail -s "success $DIR" $MailAddress <<< "successfully completed archiving $DIR"
        fi

        done

thanks guys for all your help in this script, i really do mean it, i couldnt have done it without you guys

im still really rubbish at writting scripts btw