AWS Command Line Interface (CLI) Integration
This technical module outlines the configuration and deployment of the AWS CLI toolset within high-performance computing (HPC) environments to manage scalable cloud storage fabrics directly from terminal sessions.
Core Capabilities and Use Cases
- Uploading and downloading data from Amazon S3
- Synchronizing large datasets with cloud storage
- Automating AWS workflows via scripts and SLURM jobs
Load Module and Configure Authentication
module load awscli/2.2.14
aws configure
aws configure
ℹ Note: You will be prompted for your credentials and default settings:
AWS Access Key ID: ************
AWS Secret Access Key: ************
Default region name: us-east-1
Default output format: json
AWS Secret Access Key: ************
Default region name: us-east-1
Default output format: json
Common S3 Commands
List Buckets
# List buckets
aws s3 ls <bucket name>
2025-09-01 15:12:48 my-research-bucket
2025-09-02 09:20:13 project-data-storage
aws s3 ls <bucket name>
2025-09-01 15:12:48 my-research-bucket
2025-09-02 09:20:13 project-data-storage
Copy Files (Upload/Download)
# Download from S3
aws s3 cp s3://my-bucket/file.txt .
# Upload to S3
aws s3 cp localfile.txt s3://my-bucket/path/
# Copy between buckets
aws s3 cp s3://bucket-a/file.csv s3://bucket-b/file.csv
aws s3 cp s3://my-bucket/file.txt .
# Upload to S3
aws s3 cp localfile.txt s3://my-bucket/path/
# Copy between buckets
aws s3 cp s3://bucket-a/file.csv s3://bucket-b/file.csv
Advanced Sync Options
# Dry Run (Preview changes without moving files)
aws s3 sync s3://my-bucket/project-data ~/Desktop/project-data --dryrun
# Progress View
aws s3 sync s3://my-bucket/project-data ~/Desktop/project-data --progress
aws s3 sync s3://my-bucket/project-data ~/Desktop/project-data --dryrun
# Progress View
aws s3 sync s3://my-bucket/project-data ~/Desktop/project-data --progress
ℹ Note: Sync only transfers new or changed files, making it ideal for large datasets.
Manage and Delete
# Upload directory recursively
aws s3 cp results/ s3://my-bucket/results/ --recursive
# Move file (deletes source)
aws s3 mv file.txt s3://my-bucket/file.txt
# Delete file
aws s3 rm s3://my-bucket/file.txt
# Delete folder recursively
aws s3 rm s3://my-bucket/folder/ --recursive
aws s3 cp results/ s3://my-bucket/results/ --recursive
# Move file (deletes source)
aws s3 mv file.txt s3://my-bucket/file.txt
# Delete file
aws s3 rm s3://my-bucket/file.txt
# Delete folder recursively
aws s3 rm s3://my-bucket/folder/ --recursive
⚠️ Warning: Recursive deletes cannot be undone unless versioning is enabled on your S3 bucket.
SLURM Job: AWS Transfer Example
#!/bin/bash
#SBATCH --job-name=s3_transfer
#SBATCH --partition=scu-cpu
#SBATCH --time=02:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
module load awscli/2.2.14
aws s3 sync s3://my-bucket/project-data /athena/labname/scratch/cwid
echo "Download complete"
Parallel SLURM Transfers
#!/bin/bash
#SBATCH --job-name=parallel_s3
#SBATCH --array=1-10
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=12:00:00
#SBATCH --output=logs/s3_%A_%a.log
module load awscli/2.2.14
SUBSET_DIR="/path/to/data/subset_${SLURM_ARRAY_TASK_ID}"
BUCKET="s3://your-bucket/subset_${SLURM_ARRAY_TASK_ID}/"
aws s3 sync ${SUBSET_DIR} ${BUCKET} --region us-east-1