Python Virtual Environments (venv)
⚠️ CRITICAL STORAGE RULE:
Do not build environments in your home directory (/home/cwid). Your environment must reside in your scratch space path: /athena/labname/scratch/[your_cwid]/ to avoid completely blocking storage allocations.
Do not build environments in your home directory (/home/cwid). Your environment must reside in your scratch space path: /athena/labname/scratch/[your_cwid]/ to avoid completely blocking storage allocations.
Quick Deployment
1. Initialize Environment
Creates the virtual environment directory structure inside your designated scratch space:
# Replace cwid and labname with your specific details
python3.14 -m venv /athena/labname/scratch/[your_cwid]/my_env
python3.14 -m venv /athena/labname/scratch/[your_cwid]/my_env
2. Activate Session
This step must be performed every time you spin up a new terminal window or log in:
source /athena/labname/scratch/[your_cwid]/my_env/bin/activate
3. Install Dependencies
Installs individual packages isolated entirely from the root system Python:
# Install specific packages manually
pip install numpy pandas scipy
# OR install a batch list from a requirements text file
pip install -r requirements.txt
pip install numpy pandas scipy
# OR install a batch list from a requirements text file
pip install -r requirements.txt
SLURM Integration
Inside your .sh batch compilation script, always call the absolute path to your activation file. This ensures compute nodes pinpoint your custom package tree:
#!/bin/bash
#SBATCH --job-name=venv_test
#SBATCH --mem=8G
source /athena/labname/scratch/[your_cwid]/my_env/bin/activate
python my_analysis_script.py
#SBATCH --job-name=venv_test
#SBATCH --mem=8G
source /athena/labname/scratch/[your_cwid]/my_env/bin/activate
python my_analysis_script.py
Replication & Sharing
Capture State: Save an exact list of your running pipeline versions to guarantee reproducibility downstream:
pip freeze > requirements.txt
Restore State: Rebuild or mirror the environment configuration flawlessly across other clusters:
pip install -r requirements.txt
Troubleshooting
| If you see... | Check this... |
|---|---|
| ModuleNotFoundError in a job | Did you append the absolute source .../activate target line explicitly inside your Slurm script configuration? |
| Disk Quota Exceeded | Execute du -sh ~ immediately to check if environment package blocks are accidentally dropping directly into home root directories. |
| Permission Denied | Ensure you are conducting third-party package installations strictly while your profile environment is actively targeted and running. |
Note: To completely drop out of a profile space during manual interactive shell work, type
deactivate.