How to Use rclone: General Overview & OneDrive Guide
This technical module outlines the configuration and deployment of the rclone command-line utility within high-performance computing (HPC) infrastructures to streamline cloud data management and storage workflows across multiple providers, with a walkthrough for Microsoft OneDrive.
Part 1: General Overview of rclone
1. What is rclone?
Often described as "rsync for cloud storage," rclone is an open-source command-line program that allows you to manage files with cloud storage products (including OneDrive, Google Drive, Amazon S3, and Box).
2. Core Capabilities and Use Cases
- Transfer data easily between local machines, HPC clusters, and cloud storage providers.
- Perform incremental backups, synchronizations (
sync), and data verification via checksums. - Mount remote cloud storage directories locally or stream files directly from the command line.
3. Copy vs. Sync Overview
rclone has two main transfer commands, and the difference matters:
| Command | What it does | Use it for |
|---|---|---|
rclone copy | Copies new and changed files. Never deletes anything at the destination. | Routine transfers and backups (recommended) |
rclone sync | Makes the destination an exact mirror of the source, deleting destination files that aren't in the source. | Only when you explicitly want a mirror |
rclone copy for routine transfers and backups. Avoid using rclone sync unless you explicitly intend to mirror directories, as sync will delete destination files that are absent from the source.4. General Core Commands
rclone listremotes
# Copy a file to a generic remote
rclone copy /path/to/local/file remote-name:destination-folder
# Sync a directory (mirrors source to destination, deleting extra destination files)
rclone sync /path/to/local/dir remote-name:destination-dir
Part 2: Using rclone with Microsoft OneDrive
5. Load Module & Configure Remote Storage
OR
ssh cwid@scu-login02.med.cornell.edu
scu-hpcc@scu-login02 $ srun -n1 --pty --partition=<partition_name> --mem=8G bash -i
scu-hpcc@c20 $ module avail rclone
scu-hpcc@c20 $ module load rclone/<version>
srun session first. Please review the Login Nodes Policy for more details.rclone config
When running rclone config, these are the exact steps and prompts you will see in your terminal:
| Prompt | Answer |
|---|---|
| region> | 1 (Microsoft Cloud Global) |
| Edit advanced config? | n |
| Use web browser to automatically authenticate? | n (the cluster has no browser; see step 6) |
| config_token> | Paste the token from step 6 |
| config_type> | 1 (OneDrive Personal or Business) |
| Choose drive to use | 0 |
| Is that okay? | y |
| Keep this remote? | y, then q to quit |
6. Authenticate OneDrive from an HPC Cluster (Headless)
The HPC login node does not need to have a graphical web browser. If web authentication fails or you are working on a headless login node, select n for remote browser authentication during setup and run the authorization command on your local computer (laptop/desktop):
A browser window will open. Sign in to your Microsoft account and authorize rclone. Rclone will provide a configuration token. Return to the HPC terminal and paste the token when prompted:
Do not share the token or your rclone configuration file with other users.
7. Protect Your rclone Configuration
The configuration contains authentication information. Check its location with:
8. Create a Destination Folder in OneDrive
rclone mkdir onedrive:HPC_Backups/Cayuga
# Verify the directory
rclone lsd onedrive:HPC_Backups
9. Transfer a Single File
Use rclone copy to transfer a file:
Keep in Mind: OneDrive Storage Quotas
Effective July 1, 2026, WCM introduced a soft storage quota of 2TB for OneDrive. While it is not a hard data cap, ITS will begin charging per TB for data exceeding the quota starting in FY28 (July 1, 2027).
Note: Scientific Computing does not manage OneDrive storage. For questions regarding your storage limits, quotas, or charges, please contact ITS.
# Example
rclone copy /athena/my-lab/scratch/user123/results.txt onedrive:HPC_Backups
The source file remains on the HPC. rclone copy does not delete files at the destination.
10. Transfer an Entire Directory
Rclone will transfer the contents of the directory while leaving the HPC source unchanged.
11. Show Transfer Progress
/athena/my-lab/scratch/user123/project1 \
onedrive:HPC_Backups/project1 \
--progress
The shorter form is:
12. Run Large Transfers in screen
Large transfers may take hours or days. Use screen so the transfer continues if your SSH connection is disconnected.
rclone copy \
/athena/my-lab/scratch/user123/project1 \
onedrive:HPC_Backups/project1 \
-P
# Detach from the screen session: Press Ctrl-A, then D
# Reconnect later: screen -r rclone-onedrive
# List active screen sessions: screen -ls
13. Resume an Interrupted Transfer
If a transfer is interrupted, rerun the same rclone copy command. Rclone will compare the source and destination and transfer files that still need to be copied.
/athena/my-lab/scratch/user123/project1 \
onedrive:HPC_Backups/project1 \
-P
14. Perform a Dry Run
Before a large or important transfer, use --dry-run to see what rclone intends to transfer without actually copying files.
/athena/my-lab/scratch/user123/project1 \
onedrive:HPC_Backups/project1 \
--dry-run
15. Check the OneDrive Destination
rclone lsd onedrive:HPC_Backups
rclone lsf onedrive:HPC_Backups/project1
rclone lsf -R onedrive:HPC_Backups/project1