-exec cmd in ksh script

Tanuka · September 16, 2010, 10:57am

Hi,

I discovered the following single-line script works very well to cp a large number of files from a source directory to a destination directory while avoiding the "argument list too large" error:

# cpmany - copy large number of files
# Takes two parameters - source dir, destination dir
# Copies ALL regular files from source dir to destination dir
# Does not traverse subdirectories in source
# Does not cp hidden files
#
find $1 -maxdepth 1 -type f -name '*' -exec cp -v {} $2 \;

I'm not fluent in the ksh, so I'm hoping someone who is can explain in detail the operation of this script.

Since the result of the find cmd is a list of filenames, does the "-exec" command actually spawn a new process for every filename in the list in order to do the cp? Is it a case of spawning a new ksh and then overlaying that new ksh with cp to do the copy of a file in the list?
If you remove the single quotes from the *, you get the "argument list too large" error. Exactly why is this?
Obviously, the "{}" that follows the cp command means 'use a name from the list' produced by "find". Where can I find more on this construct? It's pretty cool!
Why is a final "\" needed at the end of the line, just before the ";"?
I've tried running this thing dirrectly from the ksh command prompt (rather than as a script) but I get "missing argument to exec". Why?
I've looked and looked for detailed information on the exec cmd in ksh, and what I've found is so limited that I came here to get answers. Is there a difference between "exec" and "-exec" in this context?

Thanks in advance for your help. I hate 'black boxes' even if they work very well.

felipe.vinturin · September 16, 2010, 12:35pm

Follow the answers:

Each file returned from the find is copied with the cp command. I mean, it "spawns" a cp command;
Do you really need the -name argument? Try:

find $1 -maxdepth 1 -type f -exec cp -v {} $2 \ ;

I think find man pages have information about, it is just a placeholder. You could also try the "xargs" command.
It is part of the find command structure, but it means that the -exec argument finishes.
I am not sure if it is a typo, but in the command you post there is a missing space between the backslash and the semi-colon.
exec is a shell builtin, try man exec.

I hope it helps!

Regards!

jlliagre · September 16, 2010, 12:41pm

Yes

No. -exec is directly launching the "cp" command with its arguments. There is no ksh involved here.

Because the shell expands * and you have too many of them in the current directory.

In the "find" manual page and in the various web pages about it.

Because ';' is required to delimit the end of the -exec clause but ';' alone would be picked by the current shell as a command separator if not escaped by '\'.

Not sure, did you replace $1 and $2 by something useful ?

Absolutely. exec is a ksh builtin command while "-exec" is an unrelated find option.

---------- Post updated at 18:38 ---------- Previous update was at 18:36 ----------

It is needed to fulfill this requirement:

 # Does not cp hidden files

---------- Post updated at 18:41 ---------- Previous update was at 18:38 ----------

That's the other way around. There is an extra space between the backslash and the semicolon in your suggestion. That space is defeating the backslash requirement.

Tanuka · September 16, 2010, 3:57pm

jlliagre,

Your answers make perfect sense! This is very much appreciated. I'll study the "find" command.

This was sort of 'wigging me out' until you explained matters. Here in this script we have a seemingly bizarre example of connecting the output of one command to the input of another, and all without the use of a pipe! But of course "find" with its "-exec" option is doing much for us in this regard - so things make sense after all.

Thanks again!

---------- Post updated at 01:57 PM ---------- Previous update was at 10:56 AM ----------

One final question:

Given the script:

    find $1 -maxdepth 1 -type f -name '*' -exec cp -v {} $2 \;

which copies ALL regular files from $1 (source dir) to $2 (dest. dir)

How can I make this script take a file-pattern for -name instead of using '*', so that I can use the script to cp only the files I want?

          find $1 -maxdepth 1 -type f -name $2 -exec cp -v {} $3 \;

does not work if you use a wildcard like * in your 2nd parameter. What needs to be done to allow the use of a special char like * ?

felipe.vinturin · September 16, 2010, 4:05pm

Put it between double-quotes!

find $1 -maxdepth 1 -type f -name "$2" -exec cp -v {} $3 \;

Tanuka · September 16, 2010, 4:16pm

Hi Filipe,

When I use the double quotes around the positional paramter like this ("$2") I get wierd results when I run the script.

For example, if I have a directory with the following files in it:

    cpmany
    file1
    file2
    file3
    file4

and I'm cd'ed into this directory and I invoke cpmany like this:

./cpmany  .  f*  /joe/stuff/

what happens is file1 is copied to file2, with no files copied to the dest dir I specified on the command line, and after doing the one copy operation the script finishes, with no errors.

It should have copied all four files to the dest dir - but it does not do so. What is wrong with my syntax?

jlliagre · September 16, 2010, 4:29pm

That should be:

./cpmany . "f*" /joe/stuff/

Tanuka · September 16, 2010, 4:37pm

I see - but is there no way to do this without requiring the double quotes around the parameter passed to the script?

I can see people often forgetting to double-quote their parameter, with possibly terrible results!

jlliagre · September 16, 2010, 5:35pm

File name expansion is a shell feature that can be disabled with "set -f" in the user's shell but that would probably lead to more confusion.

Tanuka · September 16, 2010, 6:00pm

Perhaps I should first verify that the parameters are each quoted before I execute the find cmd - if they are not, I could terminate the script with an error message.

So I'd need to look for matching pairs of double-quotes. I should be able to do that fairly easily, I think.

jlliagre · September 17, 2010, 2:36am

That won't work. The quotes are removed by the shell before passing the parameters to the called program.
However, checking the arguments count should work as a workaround. If larger than the one expected, that would mean expansion had happen.

felipe.vinturin · September 17, 2010, 5:58am

Just one more comment on that, it happens because the shell is expanding the wildcard: "*" before starting the script.

Check below:

####################################
# Example files
####################################
# ls -1 f*
file1
file2
file3
file4
file5
####################################

####################################
# Example script
####################################
# cat myScript.sh
#!/bin/bash

argCount=1

for scriptArg in "$@"
do
        echo "Arg#: [${argCount}] - Value: [${scriptArg}]"
        argCount=`expr ${argCount} + 1`
done
echo "Find Result"
find $1 -maxdepth 1 -type f -name "$2"
####################################

####################################
# Exemple execution - No double quotes
####################################
# ./myScript.sh . f*
Arg#: [1] - Value: [.]
Arg#: [2] - Value: [file1]
Arg#: [3] - Value: [file2]
Arg#: [4] - Value: [file3]
Arg#: [5] - Value: [file4]
Arg#: [6] - Value: [file5]
Find Result
./file1
####################################

####################################
# Exemple execution - Double quotes
####################################
# ./myScript.sh . "f*"
Arg#: [1] - Value: [.]
Arg#: [2] - Value: [f*]
Find Result
./file2
./file1
./file4
./file5
./file3
####################################

I hope it helps!

Regards.

jlliagre · September 17, 2010, 9:15am

Indeed. I wrote how this expansion can be disabled in post #9 -exec cmd in ksh script Post: 302453896

felipe.vinturin · September 17, 2010, 9:21am

Sorry, I didn't see that!

Tanuka · September 17, 2010, 12:32pm

Ok guys, here's what I've ended up with, many thanks to you, because this works great:

#!/bin/ksh
#- cpmany.ksh
#
# Used to copy regular files from one directory to another
# when the number of files is so large that the regular cp
# cmd blows up with "arg list too large" error.
# 
# Parameter 1 is the source dir - use no wildcard in this parameter.
# Parameter 2 is the filename pattern - MUST DOUBLE-QUOTE if you use ANY wildcard.
# Parameter 3 is the destination dir - use no wildcard in this parameter.
#
# It's probably safest to cd into the source dir and simply provide
# a dot (.) as the 1st parameter, and then your desired filename pattern
# ENCLOSED IN DOUBLE-QUOTES as the 2nd parameter and finally
# the destination dir as the 3rd parameter. Don't forget to provide trailing slashes
# on directory names just to be safe.
#
if [[ "$#" -ne 3 ]]; then
    print "\nIncorrect number of args or an arg that has a wildcard is not double-quoted\n"
    print "        usage:  cpmany.ksh  sourcedir  \"filename-pattern\"  destdir\n"
else
    find $1 -maxdepth 1 -type f -name "$2" -exec cp -v {} $3 \;
fi