AI Cluster Job Monitoring and Management

Once your SLURM jobs are running, several commands can help you monitor status and manage jobs.

Cancel a Job

scancel <job_id>

Useful SLURM Commands

# show queue status
squeue -l

# show only jobs for a specific user
squeue -l -u <cwid>

# list partitions
sinfo

# detailed information about a job
scontrol show job <job_id>

# detailed information about a node
scontrol show node <node_name>

# list jobs run by a user in the past 7 days
sacct -u <cwid> -S $(date -d "-7 days" +%D) -o "user,JobID,JobName,state,exit"

Scheduling Considerations

The more resources you request from SLURM, the harder it may be for the scheduler to place your job quickly.

Larger jobs may run faster once started, but they may also remain queued longer.

Choosing the best resource request depends on:

• The type of workload you are running
• The amount of CPU, memory, or GPU resources it needs
• How busy the cluster is at the time of submission