BRB - Manually Archiving Data to AWS S3

Overview

Before using rclone to copy files to S3, you must configure a remote for your bucket.

Run rclone config and follow the prompts. This creates the configuration file:

~/.config/rclone/rclone.conf

Multiple remotes can be configured for different buckets, cloud providers, or systems. Each remote must have a unique name.

General rclone Configuration Notes

• Select AWS unless you are using a different S3-compatible provider
• Store credentials in the config file unless you specifically want environment-based auth
• The default endpoint is usually acceptable after selecting the region
• If required, choose AWS:KMS for server-side encryption
• Use DEEP_ARCHIVE if copying directly to GDA
• You can usually skip the advanced configuration section

If bucket creation permissions are not available, add the following line to the remote config:

no_check_bucket = true

Wasabi Configuration Example

Wasabi is an S3-compatible object storage platform that can also be configured with rclone.

rclone config

Example settings:

name> wasabi
Storage> s3
provider> Wasabi
env_auth> false
access_key_id> YOURACCESSKEY
secret_access_key> YOURSECRETACCESSKEY
region>
endpoint> s3.wasabisys.com

The resulting config file will look similar to:

[wasabi]
type = s3
provider = Wasabi
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
endpoint = s3.wasabisys.com

Rclone Command Examples

The syntax for specifying an S3 source or destination is:

<remote>:<bucket>/<path>

Example version check:

rclone --version

Copy

By default, rclone copy copies folders recursively.

Copy a local file to S3:

rclone copy foo.txt my-s3:my-bucket/test

Copy a folder from S3 to a local directory:

rclone copy my-s3:my-bucket/test test2

Useful Copy Options

• --checksum verifies copied content using checksums
• --progress displays progress during transfer
• --stats 30s prints stats every 30 seconds
• --transfers <num> controls parallel file copies

List

rclone lsf is a convenient listing command for human- and machine-readable output.

rclone lsf my-s3:my-bucket/test

Remove

On cloud storage such as S3, rclone delete removes all matching objects under the provided path.

rclone delete my-s3:my-bucket/test

A useful option for delete operations:

• --dry-run shows what would be deleted before performing the action