Job Arrays

Job arrays are the best way to submit many similar jobs that differ only by an index, input file, parameter, or sample. Slurm arrays are built specifically for this pattern.

When to Use Job Arrays

• One script per sample
• Parameter sweeps
• Many independent files
• Repeated preprocessing tasks

Script Examples

Example 1: One Input File per Task

#!/bin/bash -l
#SBATCH --partition=scu-cpu
#SBATCH --array=1-50
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=01:00:00
#SBATCH --output=array-%A_%a.out

INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" input_list.txt)
python process_one.py "$INPUT"

Example 2: Parameter Sweep

#!/bin/bash -l
#SBATCH --partition=scu-cpu
#SBATCH --array=1-5
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=00:30:00

PARAMS=(0.1 0.2 0.5 1.0 2.0)
VALUE=${PARAMS[$((SLURM_ARRAY_TASK_ID-1))]}

python sweep.py --alpha "$VALUE"

Useful Array Variables