HI
I want to implement a control mechanism using Shell scripts .The intention is to have controlled number of jobs running in parallel
External process will kickstart 40 jobs in parallel .All the 40 jobs will call the same generic script with different parameter values .But at a time only 2 should run ( In extended version of this i will dynamically change the value say to 5 or 10 ) .
What i tried to implement was , with in the generic script i will create a ctl file (CTL files names are unique with same extension) If the number of CTL files is more than or equal to ( -ge 2) , then i wil sleep the job .Else will run it .The problem is when 40 jobs comes back from sleep at the same time , the chances of two or process taking control is high .How can i restrict it ?
---------- Post updated 05-04-10 at 11:48 AM ---------- Previous update was 05-03-10 at 05:27 PM ----------
Any solutions ? What is the best way to implements the queuing mechanism in Shell scripts
There's probably lots of ways. One at a time is easy; anything that only one process can succeed at can be used as a locking mechanism, like mkfifo. Each process runs 'mkfifo /tmp/myfifo', if it succeeds, it has control until it deletes the fifo. if it doesn't, sleep a second then retry. Simple but not the most efficient, and not precisely a queue.
Two at a time is harder. Shell script doesn't have good mechanisms for tracking multiple children simultaneously, not in a manner to tell your script which child just terminated. It'd be possible with an arrangements of fifos to communicate between child and controller, but not trivial, and prone to breaking down when processes terminate unexpectedly.
It'd be simple enough to do and control with a makefile, if you have make.