MPI and Multi-node Jobs
MPI jobs are different from ordinary threaded jobs. For MPI, you usually scale with --ntasks, and sometimes also with --nodes or --ntasks-per-node.
MPI Scenarios
| Scenario | Recommended Request |
|---|---|
| 8 MPI ranks, any placement | --ntasks=8 |
| 8 MPI ranks across 4 nodes | --ntasks=8 --nodes=4 |
| 8 MPI ranks, 2 per node | --ntasks=8 --ntasks-per-node=2 |
| 8 MPI ranks, each with 2 CPUs | --ntasks=8 --ntasks-per-node=2 --cpus-per-task=2 |
Sample MPI Script
#!/bin/bash -l
#SBATCH --job-name=mpi_example
#SBATCH --partition=scu-cpu
#SBATCH --ntasks=8
#SBATCH --nodes=4
#SBATCH --mem=64G
#SBATCH --time=04:00:00
#SBATCH --output=mpi-%j.out
source ~/.bashrc
module purge
# load your MPI environment here
InputData=/athena/mpi/data
OutputData=/athena/mpi/outputData
rsync -a $InputData $TMPDIR
srun mpi_prog
rsync -a $TMPDIR $OutputDataSlurm works directly with MPI launch patterns, and srun is commonly used to launch MPI ranks inside the allocation.
When Not to Use Multi-node Jobs
If your application is not MPI-enabled, do not request multiple nodes just to get more cores. Most non-MPI software cannot natively utilize CPUs spread across completely different physical server nodes.