Creating script with multiple job arrays

Shell Programming and Scripting
1

Hello everyone,

First of all this is my first post and im fairly new to working with Unix and creating scripts etc. so there will probably be wrong phrases used.

Lets get to my questions.

I have multiple scripts that submit Slurms/Jobs to the cluster starting like this and doing certain tasks:

#!/bin/bash
#SBATCH --partition=
#SBATCH --ntasks=
#SBATCH --time=0-10:00
#SBATCH --mem-per-cpu=48G
#SBATCH --job-name=basic_test
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=           
#SBATCH --array=1-14

...

I have to run multiple of those scripts AFTER all the Slurms from the preceeding job are finished. Now I want to write a script that does exactly that. I already tried something like this:

#!/bin/bash

sbatch ~/testscript1.sh

wait

sbatch ~/testscript2.sh

the problem seems to be that the skript thinks that testscript1.sh is done as soon as it submitted all the slurms in the array to the server and then it starts testscript2.sh. This leads to problems because testscript2.sh works with some files that are only created when the submitted slurms from testscript1.sh are done.

Thank you for your help

2

From the man page:

So the two commands will be executed immediately after one another. The wait statement has no function here, since no shell controlled background jobs are specified/used in the shell script.

So you will need to create dependencies between jobs. Did you checkout the -d option?

1 Like
3

Thank you for the fast answer.... I didnt even think about looking into the manual -.-'
I will work with this and post results later

4

I changed the script to work with

--dependency=singleton

This works for my purpose but might cause some compications for people running different jobs

I wanted to get

 --dependency=afterok:JobID

to work but I couldnt get it to work because I couldnt aquire the JobID from the first submitted job correctly.

I tried

JOBID1=$(sbatch ~/array.job)
JOBID1=$(echo $JOBID1 | sed -e "s/[a-zA-Z ]*//")

  \# removes anything but the job id

 JOBID2=$(sbatch --dependency=afterok:$JOBID1 ~/analyzeTE2.sh)

i also tried it without the echo | sed part but it wont work either way :/. I always get:

 sbatch: error: Batch job submission failed: Job dependency problem

this shows that the jobID isnt correct but I have no clue how to extract it correctly

------ Post updated at 10:19 AM ------

Ok problem is solved

I tried to define a new variable using the same name....

I just changed it like this:

 jid1=$(sbatch -p carl.p  ~/testing/testscript1.sh)

jid2=$(echo $jid1 | sed -e "s/[a-zA-Z ]*//")

sbatch -p carl.p --dependency=afterok:${jid2} ~/testing/testscript2.sh

and it worked