Slurm Preemption
Preemption allows Slurm to reclaim compute resources from lower-priority jobs so higher-priority jobs can run. This is used on SCU clusters to improve overall utilization while ensuring fair access to dedicated lab resources.
Preemptible Partitions
Across SCU clusters (BRB, AI, and Cayuga), preemption is implemented through the following partitions:
| Partition | Resource Type | Purpose |
|---|---|---|
preempt_cpu | CPU | Run opportunistic CPU jobs on available nodes. |
preempt_gpu | GPU | Run opportunistic GPU jobs on available nodes. |
These partitions allow users to run jobs on idle resources that belong to other labs or partitions.
How Preemption Works on SCU Clusters
The SCU preemption model follows these core rules:
- Jobs in
preempt_cpuandpreempt_gpurun exclusively on otherwise idle resources. - Higher-priority jobs (e.g., from the owning lab or standard partitions) can reclaim those resources immediately.
- Preemptible jobs are completely cancelled when preempted.
Guaranteed Runtime (Important)
All preemptible jobs are guaranteed a minimum runtime of 1 hour.
- If your job has been running for less than 1 hour, it will not be preempted.
- Once your job has run for 1 hour or more, it may be cancelled at any time.
This guaranteed runtime window ensures that short jobs and test workloads can complete reliably, while still allowing efficient reuse of idle resources.
Example Scenario
Suppose you submit a job to preempt_gpu:
- Your job starts on an idle GPU node belonging to another lab.
- After 20 minutes: The owning lab submits a job → your job continues running safely because it is within the 1-hour window.
- After 1 hour: The owning lab submits a job → your job may be cancelled immediately.
Similarly, if another user's preemptible job is running on your lab’s hardware, your job may be delayed until their 1-hour minimum runtime completes.
When to Use Preemptible Partitions
Recommended for:
- Short jobs (≤ 1 hour)
- Testing and debugging workflows
- Parameter sweeps or batch experiments
- Checkpoint-enabled applications
Not recommended for:
- Long jobs without checkpointing
- Critical production workflows
- Work that cannot tolerate interruption
Submitting a Preemptible Job
CPU Example
#!/bin/bash -l
#SBATCH --job-name=preempt_cpu_test
#SBATCH --partition=preempt_cpu
#SBATCH --qos=low
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=02:00:00
python my_script.pyGPU Example
#!/bin/bash -l
#SBATCH --job-name=preempt_gpu_test
#SBATCH --partition=preempt_gpu
#SBATCH --qos=low
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --gres=gpu:1
#SBATCH --time=04:00:00
python train_model.pyBest Practices & Key Takeaways
- Design workflows assuming jobs may be interrupted after 1 hour.
- Save computational outputs and model weights frequently.
- Always use checkpointing strategies when available.
- Use standard non-preemptible partitions for long or time-critical calculations.
preempt_cpuandpreempt_gpuprovide vital access to extra cluster capacity at zero priority cost.