Bash: Regulating the number of processes a script can spawn

I've been working on some scripts in which I spawn some background processes. I'd like to be able to limit the number of processes, but have my script spawn additional processes as previous tasks finish. So, let's say I have 20 tasks to complete. Any given task could take from 1 to 10 minutes. I want to have only 3 of these background tasks going at a time, but when one completes I want another to start up until all 20 are finished. I've come up with what seems like a sloppy but effective way to do this involving "while" and "jobs", but I'd appreciate any suggestions on good ways to accomplish this general type of scripting task efficiently. How do you do this?

have them each establish any one of several specifically named lock-files...and then govern the available slots.

You could also do this quite easily with a makefile. Make has built-in mechanisms for handling multiple jobs. Here's a generic makefile that feeds ".f" files into your programs' standard input, and puts its standard output into ".g" files:

# File should be named Makefile
# Anywhere you see eight leading spaces, its actually a tab

# Tell it .f and .g are suffixes to convert
.SUFFIXES:
.SUFFIXES: .f .g

# Rule telling it how to create *.g from *.f
.f.g :
        ./myscript.sh < "$<" > "$@"

With Makefile in your current directory, when you run

make file.g

it will feed file.f into ./myscript.sh 's standard input, and create file.g from its standard output.

You can specify more than one target:

make 1.g 2.g 3.g 4.g 5.g 6.g 7.g 8.g 9.g

And you can tell it how many processes it should allow to run at once:

# Allow two simultaneous processes
make -j 2 1.g 2.g 3.g 4.g 5.g 6.g 7.g 8.g 9.g

Thanks for the replies. I've worked with lock files a little before, but never for process regulation. Good idea. The makefile option is something I've never tried, but it's really interesting. Looks like I have something new for the "to be studied" list. Again, many thanks for the suggestions.

Makefiles are more or less designed for process management. Its original, and still primary function is compiling source code but it can be applied to lots of problems as long as your program can be though of as converting one kind of data into another, repeatably(always the same input means always the same output.) Given more complicated rule relationships it can deduce the proper order to build things in, i.e. tell it how to turn .a into .b, .b into .c, and .c into .d, run 'make file.d', it will find file.a and go through all the steps to produce file.d from it.

Hi.

See also threads Parallel Execution of Programs

and

cheers, drl

This will run 100 jobs with 10 of them running at any one time:

#!/bin/bash

set -bm

startjob() {
  if [ $count -lt $total_jobs ]; then
    yourjob.sh &
    count=$(($count+1))
  fi
}

max_parallel=10
total_jobs=100

trap 'startjob' SIGCHLD

count=0
started=0
while [ $started -lt $max_parallel ]; do
  startjob
  started=$(($started+1))
done

wait

You just have to be really careful to be sure the only child processes your script spawns after the trap command are your jobs. No "ls" commands, no "rm", no "expr", nothing. Only shell built-ins.

achenle,
your approach looks very promising, but I have following questions:

1. What does "set -bm" stand for?
2. How does the flow look like? There are two variants I can think of:
   Let's assume every job will run for 5 minutes and let's forget about that SIGCHLD thing.
   2.a It starts with line 20, the function startjob is called, "started=$(($started+1))" does *NOT* take effect, 1st job gets started, $count=1, line 20 calls the startjob function again ................................. $count=100 !! All 100 jobs are running at the same time.
   2.b It starts with line 20, the function startjob is called, $started=1, 1st job gets started, $count=1, .... Script ends after 10 jobs, because $started=10.
3. IMHO that "wait" line is pretty unnecessary (pls. correct me if I'm wrong)

Is there an error in the way I think how it will work