AI Cluster Jupyter Jobs
Jupyter Notebook servers can be launched on AI Cluster compute nodes and accessed through a local browser using SSH tunneling. Because Jupyter sessions are interactive, the same efficiency considerations for interactive jobs apply here.
Prepare a Python Environment
Using Conda
Load a Python module:
module avail
module load miniconda3-4.10.3-gcc-12.2.0-hgiin2a
Create a new environment:
conda create -n myvenv python=3.10
Activate the environment:
conda init bash
conda env list
conda activate myvenv
Deactivate when needed:
conda deactivate myvenv
To disable automatic base activation on login:
conda config --set auto_activate_base false
Install Jupyter:
conda install jupyter
Using pip and venv
Load a Python module:
module avail
module load miniconda3-4.10.3-gcc-12.2.0-hgiin2a
Create and activate a virtual environment:
python -m venv ~/myvenv
source ~/myvenv/bin/activate
Install Jupyter:
pip install jupyter
Submit a Jupyter SLURM Job
Example batch script:
#!/bin/bash
#SBATCH --job-name=<myJobName>
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:30:00
#SBATCH --mem=3GB
#SBATCH --gres=gpu:1
#SBATCH -p ai-gpu
module purge
module load miniconda3-4.10.3-gcc-12.2.0-hgiin2a
# if using conda
conda activate myvenv
# if using pip
# source ~/myvenv/bin/activate
LOG="/home/${USER}/jupyterjob_${SLURM_JOB_ID}.txt"
PORT=$(shuf -i 10000-50000 -n 1)
cat << EOF > ${LOG}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~ Slurm Job $SLURM_JOB_ID
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hello from the Jupyter job!
In order to connect to this Jupyter session,
set up a tunnel on your local workstation with:
ssh -t ${USER}@ai-login01.med.cornell.edu -L ${PORT}:localhost:${PORT} ssh ${HOSTNAME} -L ${PORT}:localhost:${PORT}
Then look below for a line similar to:
http://127.0.0.1:${PORT}/?token=xxxxxxxx
Copy that line into your browser.
EOF
jupyter-notebook --no-browser --ip=0.0.0.0 --port=${PORT} 2>&1 | tee -a ${LOG}
Submit the job with:
sbatch script_name.txt
Connect to the Jupyter Session
After the job starts, it creates a log file named:
~/jupyterjob_<JOBID>.txt
Follow the instructions in that file to:
- Set up an SSH tunnel
- Open the provided
127.0.0.1:port/?token=...URL in your browser
Stopping the Jupyter Job
When finished:
- Save your work
- Close browser tabs
- Cancel the SLURM job
scancel <jobid>
⚠️ Important Reminder: Leaving idle interactive Jupyter notebooks running consumes valuable GPU cluster walltime allocations. Ensure
scancel is executed promptly following work completion.