Writing single script with mutiple sbatch parts

Shell Programming and Scripting
1

Hello,

I wrote several scripts that work with job arrays (sbatch) that look like this:

#!/bin/bash
#SBATCH --partition=carl.p
#SBATCH --ntasks=4
#SBATCH --time=0-10:00
#SBATCH --mem-per-cpu=48G
#SBATCH --job-name=basic_test
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=
#SBATCH --array=1-48

These scripts all do different things and some need to be finished before other scripts can start.

I already figured out how to write a jobdependency script that times which scripts need to be started and which need to wait until specific scripts are finished. Now I want to know if it is possible to combine all those different scripts into one big script. I want to do this because I dont want to copy all the different scripts everytime i want to use them and I also want to use the same variables in the different scripts so that I only have to define them once. This is just to make everything more user friendly.

I wanted to do something like this

#!/bin/bash
#SBATCH --partition=carl.p
#SBATCH --ntasks=4
#SBATCH --time=0-10:00
#SBATCH --mem-per-cpu=48G
#SBATCH --job-name=basic_test
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=
#SBATCH --array=1-12      



Here would be the first script that submits 12 jobs only containing the code up to the next SBATCH input...

CODE
CODE 
CODE
...




           #!/bin/bash
           #SBATCH --partition=carl.p
           #SBATCH --ntasks=4
           #SBATCH --time=0-10:00
           #SBATCH --mem-per-cpu=48G
           #SBATCH --job-name=basic_test
           #SBATCH --mail-type=END,FAIL
           #SBATCH --mail-user=
           #SBATCH --array=1-48           
           #SBATCH --dependency:afterok Job_id_of_first_script
  
            Here would be the second script which needs to run the 48 jobs from the second SBATCH input only once

            CODE
            CODE
            CODE
            ...

and so on.

These scripts should all work seperately so I dont want the first SBATCH to start a new SBATCH for the second job everytime. I have not tried anything to do this and I just wanted to know if this is at all possible.

Thanks in advance

3
Moderator comments were removed during original forum migration.
1 Like
4

Some people might have missunderstood the previous part.

I'm trying to find out if there is a way to write several sbatch scripts into one file so that this one file includes the code for all the seperate scripts.

My idea was something like this:

#!/bin/bash


variable=whatever



#SBATCH ....
Code for first sbatch 

...
end of code for first sbatch

    #SBATCH ...
    Code for second sbatch
    ...

    end of code for second sbatch
    

           #SBATCH
           Code for third sbatch
           ...
           end of code for third sbatch

and all of this should be in one big executable file including all the code and the different parameters and dependencies for the seperate sbatches.

This way I could define all the variables once and use them in every script and can make sure that the right paths for the variables are set because they are created in the scripts and saved as a variable.

I hope this makes things clearer.