Monitoring and Troubleshooting Jobs
Check Your Queue
squeue -u <cwid>Check Detailed Job Information
scontrol show jobid <job_id>scontrol is the standard tool for detailed inspection of jobs, partitions, and nodes.
Check Job History
sacct -S $(date -d "-7 days" +%D) -o "user,JobID,JobName,ReqMem,MaxRSS,NCPUS,start,end,node,state,exit"Inspect Active Job Resources
To check live resource utilization (CPU efficiency and memory footprint) for a currently running job:
sstat -j <job_id> --format=JobID,MaxRSS,AveCPUCheck Node Status
sinfoWhy Is My Job Pending?
Check the reason code listed under squeue or scontrol show jobid to self-diagnose pending states:
(Resources): The job is waiting for hardware to free up. This is normal on a busy cluster.(Priority): Other jobs submitted by you or your group have higher priority under the fairshare algorithm.(AssociationJobLimit): You have reached the maximum number of concurrent running jobs allowed for your account or project.(QOSMaxCpuPerUserLimit): Your Quality of Service (QOS) tier has capped the total number of CPU cores you can consume at once.
Slurm enforces limits through partitions, QoS, and associations, with partition-level limits often acting as hard upper bounds for time and node counts.
Troubleshooting Checklist
- Run
sinfoand confirm the partition name - Run
squeue -u <cwid>and note job state - Run
scontrol show jobid <job_id> - Review output and error files
- Check whether memory or GPU requests are larger than needed
Useful Output Files
Unless overridden, Slurm writes batch output to a file like slurm-%j.out, where %j is the job ID [schedmd.com].
Cancel a Job
scancel <job_id>