To execute scripts parallelly

Shell Programming and Scripting
1

Hi I have set two set of scripts sets in a file which perform similar operations but with different script names for e.g.:

1st set of script1.txt:
1.sh
2.sh
3.sh
4.sh

2nd set of script2.txt:

1_1.sh
2_1.sh
3_3.sh
4_4.sh

I want to execute these set of scripts parallelly in such a way that it should wait for the dependent scripts to complete before the next set of scripts run

for eg:
1.sh and 1_1.sh should run parallelly and i have maintained a completion flag in each scripts which will result like 1.completed and 1_1.completed once each scripts completes
Once both of these flags are completed next set of scripts should run parallelly
2.sh and 2_2.sh.So like this subsequent scripts should run until unless both previous scripts completes alone.
i.e 2.sh and 2_2.sh should run only once 1.completed and 1_1.completed flag is created.

2

Here's a command line I ran that demonstrates one way to do what you're talking about.

for fn in c11 c12 c13 c14; do ${fn} & done; wait; echo "First set finished"; for fn in x11 x12 x13 x14; do ${fn} & done; wait; echo "Second set finished"

The idea is to spawn background processes in batches, with "&", then use the "wait" command to pause until they are all done.

If you want to try it exactly as I ran it, the scripts are all just symlink'd to this code.

#!/bin/bash

WAIT=$(( 2 + ( ${RANDOM} % 5 ) ))

echo "Run $0 - wait ${WAIT}"

sleep ${WAIT}

echo "Done $0"

I just wanted them to finish in random order to show that "wait" really does block until all the child processes are done.

3

How about something like:

while read script1 && read script2 <&3
do
  "$script1" &
  "$script2" &
  wait
done < script1.txt 3<script2.txt

Variable script1 is read from file descriptor 0 (stdin), which is assigned to script1.txt, as defined at the bottom of the loop.
Variable script2 is read from file descriptor 3, which is assigned to script2.txt (also defined at the bottom of the loop).

1 Like
4

Hi scrut,

Can it possible to excute the scripts from particular directory path becoz each script suits belong to different path for eg 1.sh is present in $direct1 and 1_1.sh in $direct2

 
$direct1/script1 &
$direct2/script2 &
5

Sure you can, this is just the principle. You can adjust it anyway you like, for example

Direct1=/path/to/foo
Direct2=/path/to/bar

while read script1 && read script2 <&3
do
  "$Direct1/$script1" &
  "$Direct2/$script2" &
  wait
done < script1.txt 3<script2.txt
1 Like
6

Hi scrut,

In your code if any of two parallel script fails due to any reason it will not wait. i tested something like this making 1.sh executable without errors but 1_1.sh with errors. So in this case next set of scripts i.e 2.sh and 2_1.sh should not run. For this reason i maintained a flag in each scrips like touching 1.completed and 1_1.completed files until this is found you shouldn't start next set of scripts

7

Of course, it is just the principle. You can easily add code after the wait statement to test for those flags (or return codes if the scripts allow) and break the loop (with a break statement, for example) if there isn't a positive result..

8

Ok so something like this is can add right


  wait
if [[ -f $script1.completed && -f $script2.completed ]]
then 
proceed 

done < script1.txt 3<script2.txt
9

Well. Proceed is the default, so it would be a loop break, rather than proceed, something like

[[ -f "$script1.completed" && -f "$script2.completed" ]] || break
10

Ok fine how to add the name like $script1.completed becoz in my file i will have 1.sh so it should read as 1.completed rather than 1.sh.completed since the loop will hold $script1 assigned as 1.sh

11

Either change the name of the completed flag or try something like parameter expansion, "${script1%.sh}.completed"

12

I can't change the flag its the standard flag maintained but variable extension will not be touched as the flag it will 1.completed or 1_1.completed. So need to check the file with $script1.completed by taking the basename of the $script1 variable

Not getting with the below code i have deleiberity not touching completed flag in 1_1 script but rest of the set is getting executed. Considering in my script files i am not having .sh extensions

hile read script1 && read script2 <&3
do
  "./$script1" &
  "./$script2" &
  wait
[[ -f "${script1%}.completed" && -f "${script2%}.completed" ]] || break
done < s1.txt 3<s2.txt
13

Yes that is what I mean, compare:

$ foo=bar.sh
$ echo "${foo%.sh}.completed"
bar.completed
$
14

I tried something like this but the next set is running which shouldn't

while read script1 && read script2 <&3
do
  "./$script1" &
  "./$script2" &
  wait
if [[ ! -f "${script1%}.completed" ||  ! -f "${script2%}.completed" ]]
then
exit 1
fi
done < s1.txt 3<s2.txt
~
15

That parameter expansion does not look the same..

16

Note that creating flag in race condition should be done using atomic commands.

17

Can you let me know your approach ctsgnb