Monitoring and Troubleshooting Jobs
Check Your Queue
squeue -u <cwid>Check Detailed Job Information
scontrol show jobid <job_id>scontrol is the standard tool for detailed inspection of jobs, partitions, and nodes.
Check Job History
sacct -S $(date -d "-7 days" +%D) -o "user,JobID,JobName,ReqMem,MaxRSS,NCPUS,start,end,node,state,exit"Check Node Status
sinfoWhy Is My Job Pending?
Common reasons include:
- The requested partition is busy
- The job requests more memory, CPUs, or GPUs than are currently available
- The job exceeds a partition limit
- The job has lower priority than other queued jobs
Slurm enforces limits through partitions, QoS, and associations, with partition-level limits often acting as hard upper bounds for time and node counts.
Troubleshooting Checklist
- Run
sinfoand confirm the partition name - Run
squeue -u <cwid>and note job state - Run
scontrol show jobid <job_id> - Review output and error files
- Check whether memory or GPU requests are larger than needed
Useful Output Files
Unless overridden, Slurm writes batch output to a file like slurm-%j.out, where %j is the job ID [schedmd.com].
Cancel a Job
scancel <job_id>