Slurm, whose name means Simple Linux Utility for Resource Management, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters. Slurm is used at CINECA, CERN, RIKEN, LIVERMORE, OAK RIDGE, etc.
In order to run a program on the cluster, you must request resources from Slurm to generate a job. There are two ways to run a job:
interactive and sinteractive commands.When you run a batch job or an interactive job, it will be placed in a queue until resources are available.
In this chapter you will learn the basics of how to build a job file. In the next chapter more specific examples will be provided and there is a chapter dedicated to the interactive execution of jobs.
Installed Slurm version: 24.05.4 (December 2024)
In order to run a software on the cluster, the first and main option for a user is to prepare a so-called Slurm job file. A job file is a text file containing some Slurm instructions/options and the calls to the user software.
A batch job is the best option when a job can have a long or really long execution time and when it doesn't request interaction. The difference between this kind of execution and launching a program on a personal machine is that here you don't call the program you wish to launch directly but you include the call to the program inside the script which contains all that is needed to perform the computations.
A job script is a set of Linux commands and a set of resource requirements that can be passed to the Slurm job scheduler. Slurm will then generate a job according to the parameters set in the job script. Any commands that are included with the job script will be run within the job. As all the resources needed are specified inside the file, there is no need to use options on the command line.
The typical job file is something like this:
#!/bin/bash
#SBATCH --ntasks 4
#SBATCH --partition allgroups
#SBATCH --time 02:00
#SBATCH --mem 1G
cd $WORKING_DIR
#your working directory
srun <your_software>
#!/bin/bash
This is simply the interpreter of subsequent commands, bash in this case. The first directive is always on the first line of any script.
#SBATCH ...
Following the #SBATCH directive are special options geared at the Slurm scheduler. We will see some of them in detail.
cd $WORKING_DIR
This is a useful command to ask the scheduler to change the current directory before launching the program. In this way all the paths can be considered as relative to the directory the job file resides in.
srun <your_software>
This is where you ask the scheduler to run your program.
Once you've written your job file you can submit it to the scheduler to get it executed using the sbatch command:
sbatch [options] <job_file>
But first you need to connect to the login server using your credentials. Let's say your username is deiuser and your job file is called test.slurm:
ssh deiuser@login.dei.unipd.it
sbatch test.slurm
Please note
This is just a heads up, more information on how to submit a job will be provided in the next chapter: Jobs Management
When a user submits a job, he must specify among the Slurm options or flags three important elements that are:
CPUs and/or TASKs: if a user needs to run a software that is designed to use more than one CPU he has to specify the number of the CPUs that the job will use and/or if a user needs to run parallel code he has to specify the number of requested tasks;
RAM: the amount of RAM memory exclusively allocated to the applicant;
TIME: it's the expected amount of time the job will takes to end. This is mandatory and very important because if the user specifies a short amount of time that isn't enough for the job to finish, the job will be killed automatically once the specified time has elapsed resulting in an unfinished job.
Important limitations
Each job has a duration limit of 35 days: it is strongly advisable to request much less in order for the job to start in a reasonable time.
In specific cases, if one or more jobs require more execution time, you must open a ticket at the DEI Helpdesk System with the details of the request.
Warning
- If more RAM than necessary is requested, it won't be available to other users;
- if more RAM than necessary is requested, it may take a long time to have a server with the available amount;
- if less RAM than necessary is requested, the job may be unexpectedly terminated with an error.
Slurm (actually srun) recognizes a large number of options to suit your needs. Some options can be specified using either the short form (prefixed with just one dash and a letter) or the long form (two dashes and a term).
There are four mandatory options you must specify in order for your job to run on the cluster platform. These are:
-n, --ntasks
<num_tasks>
- Number of tasks to launch. For serial code <num_task> should be set to 1, for parallel code <num_task> should be set to the number of parallel execution flows.
-p, --partition
<partition_name>
- Slurm partition. For typical serial or parallel job <partition_name> is allgroups (see information in the section Partitions below)
-t, --time
<time>
- Maximum job execution time, where time could be specified as one of
- mm
- mm:ss
- hh:mm:ss
- dd-hh
- dd-hh:mm
- dd-hh:mm:ss
-m, --mem
<size[units]>
- Maximum amount of RAM memory requested. Different units can be specified using the suffix [K|M|G|T] respectively for kilobytes, megabytes, gigabytes and terabytes.
-c, --cpus-per-task <ncpus>
-w, --nodelist <node_name_list>
-x, --exclude <node_name_list>
-o, --output <filename_path>
-e, --error <filename_path>
-J, --job-name <jobname>
A complete list of all the options available for the sbatch command can be found here.
Important
Do not underestimate the importance of correctly calibrating the request of resources for a job.
Please double check if your CPUs, RAM e GPUs requests are adequate for the code you are going to execute. The efficiency must be high, otherwise your execution can result in other users having problems to launch their jobs and, in the worst case, it can lead to a forced termination of a job launched with an unwise resource allocation.
Consult the chapter How my jobs are performing for more information.
#!/bin/bash
#SBATCH --job-name <job_name>
#SBATCH --output output_%j.txt
#SBATCH --error errors_%j.txt
#SBATCH --mail-user your_email_address
#SBATCH --mail-type ALL
#SBATCH --time 02:00
#SBATCH --ntasks 4
#SBATCH --partition allgroups
#SBATCH --mem 1G
cd $WORKING_DIR
#your working directory
srun <your_software>
--job-name <job\_name>
When you queue your job this option can provide a visual clue to distinguish between your jobs.
--output output\_%j.txt
Your output file will be numbered with your JOBID (%j). Subsequent runs will not overwrite the output file.
--error errors\_%j.txt
Same as above for standard error.
--mail-user [james@gmail.com](mailto:james@gmail.com)
Depending on what you specify on the directive mail-type the specified user will be notified via email.
--mail-type <value>
Notify user by email when certain event types occur. The event list can be seen on the srun manual page on the frontend node (issuing a man srun at the command prompt) or following this link. Some possibilities are: NONE, BEGIN, END, FAIL, ALL.
The email generated by the END event is the most complete: not only it provides basic information about the job that has just terminated its execution, but it also includes the output of the seff command that shows efficiency data. So please include this type of event in order to get relevant information to better calibrate your future resource allocation: END or ALL (which includes END) both work.
A Slurm partition is a group of resources. Each partition can be considered as a job queue, each of which has some constraint (e.g. number of running jobs, job time limit, etc.).
Cluster DEI has 2 partitions:
The vast majority of the cluster users should use the allgroups partition. Typical serial and parallel software that require no interaction should be run in this partition.
The interactive partition should be used only when a real time interaction is needed and/or for tasks with low computation burden. Typical examples are the installation of software that have an interactive installation procedure or simple file managing/manipulation (e.g. compressing files).