Conda Environment Management
Conda is a powerful package and environment manager for Python and scientific computing. On HPC systems, install Miniconda in your Athena scratch directory to avoid exceeding home directory storage quotas.
ℹ Note: Replace
labname with your lab's directory name and [your_cwid] with your CWID throughout this guide.Installing Miniconda in Scratch Space
Run these commands in sequence to set up the base distribution:
# Create directory, download, and install
mkdir -p /athena/labname/scratch/[your_cwid]/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
-O /athena/labname/scratch/[your_cwid]/miniconda3/miniconda.sh
bash /athena/labname/scratch/[your_cwid]/miniconda3/miniconda.sh \
-b -u -p /athena/labname/scratch/[your_cwid]/miniconda3
# Clean up and initialize
rm -rf /athena/labname/scratch/[your_cwid]/miniconda3/miniconda.sh
/athena/labname/scratch/[your_cwid]/miniconda3/bin/conda init bash
mkdir -p /athena/labname/scratch/[your_cwid]/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
-O /athena/labname/scratch/[your_cwid]/miniconda3/miniconda.sh
bash /athena/labname/scratch/[your_cwid]/miniconda3/miniconda.sh \
-b -u -p /athena/labname/scratch/[your_cwid]/miniconda3
# Clean up and initialize
rm -rf /athena/labname/scratch/[your_cwid]/miniconda3/miniconda.sh
/athena/labname/scratch/[your_cwid]/miniconda3/bin/conda init bash
Then reload your shell and optionally update conda:
# Source your bashrc to refresh your path
source ~/.bashrc
# Update conda in the base environment
conda update -n base -c defaults conda
source ~/.bashrc
# Update conda in the base environment
conda update -n base -c defaults conda
Environment Best Practices
⚠️ Warning: Never create environments in ~/.conda/envs. Always use the --prefix flag to keep them inside your scratch storage space.
Create and activate an environment in a single workflow:
# Create environment in scratch directory
conda create --prefix /athena/labname/scratch/[your_cwid]/myenv python=3.10
# Activate using the full path
conda activate /athena/labname/scratch/[your_cwid]/myenv
conda create --prefix /athena/labname/scratch/[your_cwid]/myenv python=3.10
# Activate using the full path
conda activate /athena/labname/scratch/[your_cwid]/myenv
Moving an Existing Environment
Do not use mv directly, as it breaks hardcoded internal binary paths. Export and recreate the environment instead:
# Export existing env to a YAML file
conda activate my_old_env
conda env export > environment.yml
# Recreate the env in the new scratch location
conda env create --prefix /athena/labname/scratch/[your_cwid]/new_env -f environment.yml
conda activate my_old_env
conda env export > environment.yml
# Recreate the env in the new scratch location
conda env create --prefix /athena/labname/scratch/[your_cwid]/new_env -f environment.yml
Quick Checklist
| Rule | Target Action |
|---|---|
| ✅ Installation | Install Miniconda directly to /athena/.../miniconda3 |
| ✅ Environments | Always explicitly use the --prefix flag for new profiles |
| ✅ Home Space | Keep /home/cwid reserved for light profile config files only |
| ✅ Maintenance | Run conda clean --all periodically to free up unused storage block allocations |