sup3

A standalone S3 tool

Warning

This is alpha

Anything and everything could go wrong. The Rust AWS SDK used is also in developer preview.

Usage goal

• CI and other minimal environments where pulling in python or building dependencies might be undesirable

• Eventually, as a consistent, reliable, fast and fun S3 tool for interactive use

Design

• Act as much as possible like existing unix commands (e.g. cp, ls, cat)
• Progress reporting by default where it makes sense (e.g. cp, down, up)

Features

• Cross-platform

• Uses authorisation support from the Rust AWS SDK

• Streaming async transfers (files not read into memory)

• Upload resume on remote errors (provided by the SDK)

• No startup delay

• No runtime non-platform dependencies (e.g. libc, libm, libgcc_s)

• Concurrent transfers

• Optional progress reporting

• Recursive upload support

• Recursive download support

• Remote globbing (e.g. sup3 ls s3://bucket/media/**/highres*.png .)

• Binary size reduction

• Upload arguments, e.g. ACLs

• Config file support

• List only files or only directories

• List paging

• Server to server copy

• Custom endpoints for other S3-compatible hosts (--endpoint)

• A sync operation (as part of copy/up/down or separate)

Commands

• Upload (upload|down) (local files to S3 remote)
• Download (download|down) (S3 URIs to local file/directory)
• Remove (rm) (S3 URIs)
• List (ls) (1..N S3 URIs)
• List Buckets (list-buckets|lb)
• Copy (cp)
• Cat (S3 URIs)
• Make Bucket (mb) (S3 URIs)

Speed

Meaurements performed on an i7-7700HQ, with an approximately 50ms RTT to the S3 server.

Download - 20MiB file

Download - 224byte file

ls - 63,186 files from bucket by substring