Parallel processing and error checking

Shell Programming and Scripting
1

Hi,
I need to run multiple scripts in parallel. The scripts needs to ensure that x scripts are always running. I trigger 6 jobs in parallel, as soon as one finishes it triggers the next one, but id something fails it should stop processing any further jobs. I was able to get hold of a script which helped me in first part, but not sure how to handle the error tracking. I do not have GNU parallel :frowning:

#!/bin/ksh
PROCS=6
SLEEP=20
while read LINE
do
     while true
     do
         NUM=$(jobs | wc -l)
         echo NUM=$NUM
         if [ $NUM -lt $PROCS ]
         then
             stuff "$LINE" &
             break
         else
             sleep $SLEEP
         fi
    done
done < textfile
# wait for ALL remaining processes
wait

So above ensures 6 processes are always running. I want a way to find out if one of the process is failed, then do not trigger anymore jobs.

Thanks

2

Hi.

This is a feature of GNU parallel :

       --halt-on-error <0|1|2>
       --halt <0|1|2>
       -H <0|1|2> (-H will be retired 20120122)
                0  Do not halt if a job fails. Exit status will be the number
                   of jobs failed. This is the default.

                1  Do not start new jobs if a job fails, but complete the
                   running jobs including cleanup. The exit status will be the
                   exit status from the last failing job.

                2  Kill off all jobs immediately and exit without cleanup. The
                   exit status will be the exit status from the failing job.

See man parallel for details. If GNU parallel is not in your repository, see GNU Parallel - GNU Project - Free Software Foundation

Best wishes ... cheers, drl

3

I have mentioned in my post I do not have GNU parallel

---------- Post updated at 01:20 PM ---------- Previous update was at 09:02 AM ----------

Is there a way to capture pids of each process triggered in array and check for the exit status. For non-0 exit status I will stop triggering jobs

4

If you are allowed to run your own perl script you are allowed to run GNU Parallel.

See: _: Excuses for not installing GNU Parallel

5

HI,
I have worked with parallel in earlier projects. In my current project the Admins will not install GNU parallel. So is there a way out with the code I have written?

Thanks

6

I am confused now. Did you read _: Excuses for not installing GNU Parallel? And did you understand that you do *not* need admins to install GNU Parallel as long as you are allowed to run your own scripts (excuse #3)?

If yes: Please elaborate why none of the excuses apply to you.

7

Run a controll shell that can write an error file:

errfile="errorfile"
rm "$errfile"
while read LINE && [ ! -f "$errfile" ]
do
...
  while [ ! -f "$errfile" ]
  do
...
    (stuff "$LINE" || echo "bad exit status" > "$errfile") &
...
8

Admins had blocked access to blogs from work location.Got home and read point 3.Will try it out tomorrow