Two questions on find with rm command

Version: Oracle Linux 6.4

In the below directory, we had 1.6 million audit log files with the extention .aud which are older than 20 days .

I ran a find with rm command as shown below. But, I had to cancel the execution of the below command after 4 hours as I don't want to run a long running command during peak hours

find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20 -exec rm -f {} \;

Question1.Is there a way I could tweak this command to run faster ?

This server has no shortage of CPUs. It has 40 CPU cores ( 80 cores when Hyperthreading is considered ). It has 256G RAM too.

Later I realized that this directory has files which are older than 500 days. So, I should have done this in chunks ; 100 at time as shown below.

find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +500 -exec rm -f {} \;
find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +400 -exec rm -f {} \;
find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +300 -exec rm -f {} \;
find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +200 -exec rm -f {} \;
find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +100 -exec rm -f {} \;
find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20 -exec rm -f {} \;

So, I ran the following command to delete files older than 500 days. But it still took 32 minutes to delete 76,000 files !!

$ find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +500 | wc -l
76099
$ find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +500 -exec rm -f {} \;

So, I would like to know if there is a way I could tweak this command to run faster ?

Question2.

find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20 -exec rm -f {} \;

During the above mentioned find and rm command execution , from another session, I tried to run an ls command on this directory to see how many files are remaining. But, I got the below error

$ pwd
/u01/product/11.2.0/rdbms/audit
$ ls -alrdt *.aud | wc -l
-bash: /bin/ls: Argument list too long
0

Is there a way I could see the progress of this command with something like a progress bar?

Hello John,

There are 2 points here.
1st: You can use following command which will be faster than current command.

 find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20 -exec rm -rf {} +
 

2nd: If you want to see progress of command, means which content has deleted you can use following command, but yes it will slower than above because it is using while with find .

 find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20  | while  read file; do rm -rf $file;if [[ $? == 0 ]]; then echo $file " has been deleted"; else echo $file " has NOT been deleted"; fi;done
 

Thanks,
R. Singh

The + , you will benefit the most, since rm command will be executed for group of files find finds, opposing to \; which will work one by one.

Linux find has a switch -delete .
You might check performance running that (should be faster since no external program is passed to exec, in your case rm).

Thank You Ravinder, Peasant

Following is from the man page of find in Oracle Linux 6.4

 -exec command {} +
              This  variant  of the -exec action runs the specified command on
              the selected files, but the command line is built  by  appending
              each  selected file name at the end; the total number of invoca-
              tions of the command will  be  much  less  than  the  number  of
              matched  files.   The command line is built in much the same way
              that xargs builds its command lines.  Only one instance of  �{}'
              is  allowed  within the command.  The command is executed in the
              starting directory.

This is what I understand from the above paragraph of find's man page.

When you use -exec rm -rf {} \; , the rm command is executed for each file (making it slower) .
and when you use -exec rm -rf {} + , the rm command is executed once in a while (although the frequency of rm execution is not mentioned in man page).
Is my assumption right ?

Hello John,

-exec...\; will run one item after another. So if you have three files, the exec line will run three times. -exec ... {} + is for commands that can take more than one file at a time (eg cat, stat, ls ). The files found by find are chained together like an xargs command. This means less forking out and for small operations, can mean a substantial speedup.

 $ mkdir testdir
$ touch testdir/{0000..9999}
 $ time find testdir/ -type f -exec cat {} \;
real    0m8.622s
user    0m0.452s
sys     0m8.288s
 $ time find testdir/ -type f -exec cat {} +
real    0m0.052s
user    0m0.015s
sys     0m0.037s
 

Thanks,
R. Singh

This is basically correct. Notice that you can delete several files at once because rm takes not a single file name but a file list as an argument. Suppose you have 4 files, "a", "b", "c" and "d" you could use:

rm -f a b c d

and have them deleted in one call of rm . This is why a call like

rm -f *

works: the shell will expand "*" to such a list of files prior to even call rm and it will happily take it.

On the other hand, command lines have a limited length and the aforementioned "*" might make the command fail once there are too many file names it expands to. Furthermore, every command can only take so many arguments. You may want to try this (in a non-destructive way): execute

ls *

in the directory with the 1.6 million files of yours you will perhaps see either a "command line too long" or a "too many arguments" error. The same would happen with rm for the same reason.

So this is why creating such a list by find and then feed it to a program (regardless of this program being rm or something else) is a bad idea. This is why the command xargs was developed and for the same reason there is the "+" device in find . Both these are designed to cut a big, unmanageable list into smaller pieces and feed these pieces to a program, one at a time.

I hope this helps.

bakunin

Thank You very much Bakunin, Ravinder

Question3.

I ran an ls command from within find using \; and + variants as shown below.
Both seems to return same results. If \; variant is slow , then why do people even use it ?

[root@emeatst179 test3]# ls -alrdt
drwxr-xr-x. 2 root root 20480 Jun 26 13:04 .
[root@emeatst179 test3]#
[root@emeatst179 test3]# touch {0..9}
[root@emeatst179 test3]#
[root@emeatst179 test3]# find /tmp/test3 -exec ls -alrdt {} \; | wc -l
11
[root@emeatst179 test3]# find /tmp/test3 -exec ls -alrdt {} + | wc -l
11
[root@emeatst179 test3]# find /tmp/test3 -exec ls -alrdt {} +
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/9
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/8
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/7
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/6
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/5
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/4
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/3
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/2
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/1
-rw-r--r--. 1 root root     0 Jun 26 13:04 /tmp/test3/0
drwxr-xr-x. 2 root root 20480 Jun 26 13:04 /tmp/test3
[root@emeatst179 test3]#
[root@emeatst179 test3]#
[root@emeatst179 test3]# find /tmp/test3 -exec ls -alrdt {} \;
drwxr-xr-x. 2 root root 20480 Jun 26 13:04 /tmp/test3
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/3
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/5
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/8
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/2
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/1
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/0
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/9
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/4
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/7
-rw-r--r--. 1 root root 0 Jun 26 13:04 /tmp/test3/6
[root@emeatst179 test3]#

Question4.
Ravinder's quick test shows that \; variant takes 8 seconds and + variant takes less than a second. Significant improvement.

So, let me rephrase my interpretation of the excerpt from find's man page which I pasted above.

When we use -exec rm -rf {} \; , the rm command is executed for each file .
When we use -exec rm -rf {} + , the rm command will process several files in each execution.

For example: If you have 5 files in a directory named a b c d e

-exec rm -rf {} \; variant will be executing 5 times
So, internally it will be executing something like below

-exec rm -rf {a} \;
-exec rm -rf {b} \;
-exec rm -rf {c} \;
-exec rm -rf {d} \;
-exec rm -rf {e} \;

-exec rm -rf {} \; variant will be executing maybe once for every 5 files. So, internally it will be executing something like below

-exec rm -rf {a,b,c,d,e} \;

Regarding how often rm command will be executed when you use -exec rm -rf {} + variant , the documentation is not very clear. It just says " the total number of invocations of the command will be much less than the number of matched files " as shown above

Are my above assumptions correct ?

-delete should be even faster than -exec rm -f {} +
I suggest to first print

find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20 -print

then print and delete

find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20 -print -delete

--
Your assumptions are correct.
If you want to test how often a command is run then test with -exec echo {} + .

find / -xdev -exec echo {} + | tr -cd ' \n' | awk '{print NR". invocation with",length,"args"}'

You will find that it can run one echo with several hundred arguments. (Several thousands on most Unix OS.)

Although it isn't in the standards, many mv utilities have a -v (verbose) option that will print the names of file being deleted as they are processed. So:

find /u01/product/11.2.0/rdbms/audit -name '*.aud' -mtime +20 -exec rm -fv {} +

will show you how quickly it is progressing. And, if you change that to:

cd /u01/product/11.2.0/rdbms/audit
find . -name '*.aud' -mtime +20 -exec rm -fv {} +

it will run a lot faster because with shorter pathnames, rm can be given more files to process on each invocation.

Even if you use -delete instead of -exec rm ... , it will run faster if you're in the directory you're processing since each unlink(pathname) call performed by find or rm won't have to search the root, u01 , product , 11.2.0 , and rdbms directories to find each file it wants to remove.

Thank You MadinGermany, Don

People use it if external utilities cannot accept multiple arguments or if they need to parse line by line in while / for loops.

Also, a good advice is to organize your files in directories.
Having millions of files in one directory will never be fast.

Hope that clears things out.
Regards
Peasant.

There are also some commands where an argument must be specified after the argument that names a file. With \; you can place command args after the selected path:

find dir(s) primaries -exec cmd earlier args {} later args \;

but with + , the argument specifying the pathname(s) must be at the end:

find dir(s) primaries -exec cmd earlier args {} +

For example, the standard mv utility does not have a -t destination option. So on systems that don't have that extension, you can't use find directly to move multiple files to a single destination directory in a single invocation of mv . You have to move one file at a time:

find . -type f -name 'a*' -exec mv {} directoryA \;

or create an intermediary script to set the desitination directory appropriately:

find . -type f -name 'a*' -exec mv_back directoryA {} +

where the mv_back shell script is something like:

#!/bin/ksh
dest="$1"
shift
mv "$@" "$dest"

Are we having fun yet?

If you fear that the long running find steals too much i/o bandwith then you can use ionice (works like nice but is focused on i/o)

ionice -c 2 -n 7 find ...