I suppose that there is a question about how do you want tit to run. Is it that you want them all in straight away but with no more that 10 running (presumably to manage the load/contention) or do you want to run in blocks of 10 where all 10 have to finish before you start the next block?
For instance if you have a job that just went to sleep for the number of seconds of it's job number, (1, 2, 3, 4, .... etc.) Would you want logic A or B?
Logic A
00:00:00 Start job 1
00:00:00 Start job 2
00:00:00 Start job 3
00:00:00 Start job 4
00:00:00 Start job 5
00:00:00 Start job 6
00:00:00 Start job 7
00:00:00 Start job 8
00:00:00 Start job 9
00:00:00 Start job 10
00:00:01 Ended job 1
00:00:02 Ended job 2
00:00:03 Ended job 3
00:00:04 Ended job 4
00:00:05 Ended job 5
00:00:06 Ended job 6
00:00:07 Ended job 7
00:00:08 Ended job 8
00:00:09 Ended job 9
00:00:10 Ended job 10
00:00:10 Started job 11
00:00:10 Started job 12
00:00:10 Started job 13
:
:
etc.
Logic B
00:00:00 Start job 1
00:00:00 Start job 2
00:00:00 Start job 3
00:00:00 Start job 4
00:00:00 Start job 5
00:00:00 Start job 6
00:00:00 Start job 7
00:00:00 Start job 8
00:00:00 Start job 9
00:00:00 Start job 10
00:00:01 Ended job 1
00:00:01 Started job 11
00:00:02 Ended job 2
00:00:02 Started job 12
00:00:03 Ended job 3
00:00:03 Started job 13
00:00:04 Ended job 4
00:00:04 Started job 14
:
:
etc.
Logic A will work if you are sure you have no other background processes and 'simply':-
for single_job in ${job_list}
do
"${single_job}" &
((active_count=$active_count+1))
if [ $active_count -ge 10 ]
then
wait # All background processes must end before we start the next block
active_count=0 # Reset the counter
fi
done
echo "Finished them in blocks or up to ten"
If you need logic B then you need to keep a track of them more carefully. You can use clever data holding processes to keep track of a specific process id, or if they will be unique enough, just count the running processes:-
for single_job in ${job_list}
do
active_jobs=$(ps -f|grep -c slee[p]) # This is the important bit!
if [ $active_count -ge 10 ]
then
echo "All running at full capacity"
sleep 1 # Pick a suitable check frequency
else
"${single_job}" &
fi
done
echo "Finished them with up-to-ten running"
The important bit
You need to consider
- what you are searching for. Get it wrong and you might not submit anything or you might submit everything in one go.
- how to make sure you don't find the grep you are searching with as a process itself else you are using up one of your planned ten background jobs
For consideration a, my example I just use a sleep, so you can see it in what I've coded. You need to be sure that you find the right things and that your 'single jobs' don't actually spawn sub-processes of the same name else you will count them twice. You could look for sub-processes of your running script if that is important to you.
For consideration b, I have set the search with an expression by wrapping a character in [
& ]
This means that the grep will expand this to any string matching the expression (for which there is only one option, of course) but crucially it will ignore itself, because the expression doesn't match the square bracket itself.
Do either of these approaches help?
Kind regards,
Robin