Jobs and SLURM

Last updated Jun 10, 2026

AI Cluster Jobs and SLURM

Computational jobs on the AI Cluster are managed through SLURM. A full SLURM tutorial is available separately. This page provides practical examples that are immediately applicable on the AI Cluster.

Running application code directly on login nodes is prohibited. Login nodes are shared resources intended for light tasks such as file management, environment setup, and job submission. Heavy computations on login nodes can negatively affect other users. All computational work should be submitted through the SLURM scheduler.

Batch vs Interactive Jobs

Batch Jobs

Batch jobs are the preferred way to run computations on the AI Cluster. They allow the scheduler to manage resources efficiently and maximize cluster utilization.

Interactive Jobs

Interactive jobs are sometimes necessary, but they are less efficient because they reserve compute resources while waiting for user input. Whenever possible, users should run computations in batch mode so cluster resources can be used more effectively across the research community.

Job Preemption

Understanding Preemptible Jobs: Job preemption allows otherwise idle nodes to be used by researchers outside the owning lab. This improves overall cluster utilization. Preemptible jobs are best suited for: quick tests and debugging, short computations, and jobs that can checkpoint and resume after interruption. These jobs may be cancelled if the owning lab needs its resources back.

How to Submit a Preemptible Job:

#SBATCH --partition=preempt_cpu
# or
#SBATCH --partition=preempt_gpu

#SBATCH --qos=low

The partition selects cluster-wide preemptible resources, and the QoS lowers priority so the job can use idle capacity without interfering with higher-priority lab jobs.

PreemptExemptTime: Preemptible jobs are guaranteed a minimum run time before they can be cancelled. This minimum protected period is called PreemptExemptTime. It is currently set to 30 minutes. After that threshold is reached, the job may be cancelled at any time if a higher-priority job needs the node.

Code Example

The example below estimates the value of π using a Monte Carlo method in C with OpenMP.

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <omp.h>

long long monte_carlo_pi(long long num_samples, int num_threads) {
    long long inside_circle = 0;
    #pragma omp parallel num_threads(num_threads)
    {
        unsigned int seed = 1234 + omp_get_thread_num();
        long long local_count = 0;
        #pragma omp for
        for (long long i = 0; i < num_samples; i++) {
            double x = (double)rand_r(&seed) / RAND_MAX;
            double y = (double)rand_r(&seed) / RAND_MAX;
            if (x * x + y * y <= 1.0) {
                local_count++;
            }
        }
        #pragma omp atomic
        inside_circle += local_count;
    }
    return inside_circle;
}

int main(int argc, char *argv[]) {
    if (argc != 3) {
        printf("Usage: %s <num_samples> <num_threads>\n", argv[0]);
        return 1;
    }

    long long num_samples = atoll(argv[1]);
    int num_threads = atoi(argv[2]);

    double start_time = omp_get_wtime();
    long long inside_circle = monte_carlo_pi(num_samples, num_threads);
    double end_time = omp_get_wtime();

    double pi_approx = 4.0 * (double)inside_circle / num_samples;

    printf("Approximated π: %.15f\n", pi_approx);
    printf("Error: %.15f\n", fabs(pi_approx - 3.141592653589793));
    printf("Execution Time: %.6f seconds\n", end_time - start_time);

    return 0;
}

Compile code.c with:

gcc -fopenmp code.c

This produces an executable named a.out. Example run:

./a.out 1000 8

SLURM Batch Job Example

#!/bin/bash
#SBATCH --job-name=<jobname>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --time=00:30:00
#SBATCH --mem=8GB
##SBATCH --gres=gpu:1
#SBATCH -p <partition_name>

cd <code_directory>
export OMP_NUM_THREADS=4
./a.out 1000000000 4

Submit the job with:

sbatch script_name

View your queued jobs with:

squeue -u <cwid>

SLURM Interactive Job Example

Interactive jobs are not recommended for routine computation, but they are sometimes needed.

srun --nodes=1 \
    --tasks-per-node=1 \
    --cpus-per-task=4 \
    --partition=<partition_name> \
    --gres=gpu:1 \
    --pty /bin/bash -i

Once started, you will be placed in an interactive shell on a compute node. When finished, run exit to end the shell and cancel the interactive job.

HPC Documentation