Keep a certain number of background processes running

I've got a bit of code I'm trying to work on...
What i want to happen is ... at all times have four parallel mysql dump and imports running.

I found the follow code snippet on the forum and modified it to work
by starting four concurrent processes but it waits until all four are done before starting the next four...

 
LIST='wmi_product wmi_patches employee actions jobs appversions os hardware detection_apps status wmi_networkadapterconfig wmi_logicaldisk'
let counter=0;
for i in $LIST
do
   echo "    Dumping and Importing table $i" " ( `date +'%x %X'` ) "
   $MYSQLDUMP -h$DBHOST -u$USER -p$PASSWORD -q --opt --single-transaction --add-drop-table $DB --tables $i | $MYSQL -u$DBuser -p$DBpass $DB > $ilog.txt &
    let counter=$counter+1
    echo "$counter%4" | bc
    if [ `echo "$counter%4" | bc` -eq 0 ] ; then
          wait
    fi
done;

Is there a way to do this?
Any help would be appreciated.

I don't see any procedure to decrement your counter variable once inside the loop. So if one process stops, the counter should decrement by 1 and then you would launch another process.

But what confuses me, is how does your script ever get out of the 'if' test once you've reached four concurrent processes?

Also, you should probably set the counter by looking at how many MYSQL processes you have running, and not just counting how many have started. This way you automatically have the current number of processes needed.

So you could use (depending upon shell):

counter=`ps -f | grep -c $MYSQLDUMP`

I had a similar problem - submit up to and including four background jobs, and when one or more finish, built back up to the max.

while read LINE                          # go through the output file from the view one line at a time
do

### Commented a lot of code - but basically, based on the LINE, 
###   set up the job with export statements

###   the secret is the jobs command gives a list of background jobs to this process
###   then the wc -l tells me how many

###   if it is >= 4, it goes to sleep for 30 seconds, then checks the count again

    # getting here, we want to submit the job to run, up to the limit of 4 jobs running at one time
    # See if there are 4 jobs running right now
      while [[ $( jobs | wc -l ) -ge 4 ]]
      do
        echo_f going to sleep now at $(date)
        sleep 30
      done

      /avidyn/scripts/avd_runit_002.ksh 1>/dev/null 2>&1 &   # parameters are passed by the export statements

done < /avidyn/tmp/avd_runit_001_jobs_to_run.txt

###  Not show after the done, is a "wait" command, to wait until all background jobs are finished.

 
echo $$ >/tmp/caller.pid
HOSTFROM=192.168.0.3
HOSTTO=192.168.0.2
DBFROM=hronline
DBTO=hronline
DBUSER=admin
DBPASSWD=password
TABLES1=`mysql -u$DBUSER -p$DBPASSWD -h $HOSTFROM -e "SELECT table_name as 'removebeforeuse' FROM information_schema.tables WHERE table_schema='$DBFROM';" |grep -v removebeforeuse`
TABLES=( ${TABLES1} )
CALLERPID=`cat /tmp/caller.pid`
COUNTER=0
while [ $COUNTER -lt ${#TABLES[@]} ]; do
    NUMPROC=` ps -ef | grep $CALLERPID | grep -v grep | grep -v bash | wc -l `
    if [ $NUMPROC -lt 5 ]; then
        echo Dumping table $COUNTER of ${#TABLES[@]} -- ${TABLES[$COUNTER]} Processes running concurrently: $NUMPROC
        mysqldump -v -u$DBUSER -p$DBPASSWD -q -h $HOSTFROM -e -C $DBFROM ${TABLES[$COUNTER]} | mysql -u$DBUSER -p$DBPASSWD -h  $HOSTTO $DBTO &
        sleep 3;
        let COUNTER=COUNTER+1
    fi
done
rm -f /tmp/caller.pid
 

Another way you can try to check is:

1. When you launch a job in background, the pid is in $!
2. Append this to running comma-sep list of PIDS.
3. Instead of checking using ps -ef |grep etc. (which is prone to errors due to false match etc.+more resource intensive), do ps -o pid="" -p <pid_list>

e.g

$ ps -o pid="" -p 7693,3351,25875,9999 |wc -w
     2

So you have the count of processes alive in background in a more accurate and inexpensive manner.

Replace by:

while [ $(ps ax|grep -ic $MYSQLDUMP) -gt 4 ] ;do sleep 3;done

While 4+1(grep) sleep 3 seconds , or you can use pgrep but that's another story.

I found this script in Go deh!: Batch Process Runner in bash shell

  1 #!/bin/bash
  2 
  3 ##
  4 ## process_runner.sh <concurrent> <total procs>
  5 ##
  6 ## Example script given the maximum number of processes to
  7 ## run concurrently, c, and the total number of processes, n
  8 ## runs at most c, processes in the background until all n
  9 ## Have been run.
 10 ##
 11 ## Author Donald 'Paddy' McCarthy Dec. 17 2007
 12 ##
 13 
 14 # how many processes to run in parallel
 15 concurrent=$1
 16 # total processes to run
 17 maxprocs=$2
 18 
 19 printf "\n## STARTING %i Processes, %i at a time\n\n" \
 20   $maxprocs $concurrent
 21 
 22 
 23 # main loop wait time between checking background procs.
 24 tick=1
 25 
 26 
 27 # dummy processes sleep for a random time
 28 function runproc {
 29   local -i n=$1
 30   local -i j
 31   (( j = 5+$RANDOM*10/32767 ))
 32   #(date; printf "#>>>JOB %i : Sleeping for %i\n" $n $j)
 33   printf "OUT JOB ,%03i, Sleep for ,%2i,  , @,%s\n" $n $j "`date`"
 34   sleep $j
 35   returned=$?
 36   printf "IN  JOB ,%03i, Slept for ,%2i, returned ,%i, @,%s\n" \
 37     $n $j $returned "`date`"
 38   #(date; printf "#<<<JOB %i : Slept for %i\n" $n $j)
 39 }
 40 
 41 function print_runstats {
 42   printf '# %i Jobs in background. %i/%i started\n\n'  \
 43     `jobs -r|wc -l` $ran $maxprocs
 44 }
 45 
 46 # Bash array running keeps track of the background process numbers
 47 # Start with nothing running (sentinel value will not be a process number
 48 for ((i=0; i<$concurrent; i+=1 )); do running[$i]=123456789; done
 49 
 50 ran=0
 51 until
 52   while [ $ran -lt $maxprocs ]; do
 53     for ((p=0; p<$concurrent; p+=1 )); do
 54       proc=${running[$p]}
 55       # Over all running processes...
 56       # $proc still running?
 57       ps -p $proc | fgrep $proc >/dev/null
 58       if [ $? -ne '0' ] ; then
 59         # Not found  i.e. finished
 60         # Run another in background and store the proc number
 61         runproc $ran &
 62         running[$p]=$!
 63         ((ran+=1))
 64         if [ $ran -ge $maxprocs ]; then break 1; fi
 65       fi
 66     done
 67     sleep $tick
 68     # Status
 69     print_runstats
 70   done
 71 
 72   sleep $tick
 73   # Status
 74   print_runstats
 75 do [  `jobs -r|wc -l` -eq 0 ]
 76 done
 77 wait
 78 printf "\n## FINISHED\n"
 79 
 80 exit 0
 81 
 82 sample_output=<<!
 83 bash$ ./process_runner.sh 2 5
 84 
 85 ## STARTING 5 Processes, 2 at a time
 86 
 87 OUT JOB ,000, Sleep for ,10,  , @,Tue Dec 18 09:26:00 GMTST 2007
 88 OUT JOB ,001, Sleep for , 8,  , @,Tue Dec 18 09:26:00 GMTST 2007
 89 # 2 Jobs in background. 2/5 started
 90 
 91 # 2 Jobs in background. 2/5 started
 92 
 93 # 2 Jobs in background. 2/5 started
 94 
 95 # 2 Jobs in background. 2/5 started
 96 
 97 # 2 Jobs in background. 2/5 started
 98 
 99 # 2 Jobs in background. 2/5 started
100 
101 IN  JOB ,001, Slept for , 8, returned ,0, @,Tue Dec 18 09:26:09 GMTST 2007
102 # 1 Jobs in background. 2/5 started
103 
104 OUT JOB ,002, Sleep for ,11,  , @,Tue Dec 18 09:26:09 GMTST 2007
105 IN  JOB ,000, Slept for ,10, returned ,0, @,Tue Dec 18 09:26:10 GMTST 2007
106 # 2 Jobs in background. 3/5 started
107 
108 OUT JOB ,003, Sleep for , 7,  , @,Tue Dec 18 09:26:11 GMTST 2007
109 # 2 Jobs in background. 4/5 started
110 
111 # 2 Jobs in background. 4/5 started
112 
113 # 2 Jobs in background. 4/5 started
114 
115 # 2 Jobs in background. 4/5 started
116 
117 # 2 Jobs in background. 4/5 started
118 
119 IN  JOB ,003, Slept for , 7, returned ,0, @,Tue Dec 18 09:26:18 GMTST 2007
120 # 1 Jobs in background. 4/5 started
121 
122 OUT JOB ,004, Sleep for , 6,  , @,Tue Dec 18 09:26:19 GMTST 2007
123 # 2 Jobs in background. 5/5 started
124 
125 IN  JOB ,002, Slept for ,11, returned ,0, @,Tue Dec 18 09:26:20 GMTST 2007
126 # 1 Jobs in background. 5/5 started
127 
128 IN  JOB ,004, Slept for , 6, returned ,0, @,Tue Dec 18 09:26:25 GMTST 2007
129 
130 ## FINISHED
131 !
132 
133 
134 

All credits go to the devoloper.
I am using it.
it is working flawlessly.