cloud-copy

A library and CLI tool for copying files to and from cloud storage.

Request Feature · Report Bug · ⭐ Consider starring the repo! ⭐

📚 About cloud-copy

cloud-copy is a library and command line tool for copying files to and from cloud storage.

Supported Cloud Services

In addition to the supported cloud services below, cloud-copy supports unauthenticated downloads over HTTP.

Azure Blob Storage

Supported remote URLs for Azure Blob Storage:

• az schemed URLs in the format az://<account>/<container>/<blob>.
• https schemed URLs in the format https://<account>.blob.core.windows.net/<container>/<blob>.

If authentication is required, the URL is expected to contain a SAS token in its query parameters.

AWS S3

Supported remote URLs for S3 Storage:

• s3 schemed URLs in the format: s3://<bucket>/<object> (note: uses the default region).
• https schemed URLs in the format https://<bucket>.s3.<region>.amazonaws.com/<object>.
• https schemed URLs in the format https://<region>.s3.amazonaws.com/<bucket>/<object>.

If authentication is required, the following environment variables may be set when invoking cloud-copy:

• AWS_ACCESS_KEY_ID - the AWS access key ID to use for authentication.
• AWS_SECRET_ACCESS_KEY - the AWS secret access key to use for authentication.
• AWS_DEFAULT_REGION - the default region to use for any s3:// URLs.

Google Cloud Storage

Supported remote URLs for Google Cloud Storage:

• gs schemed URLs in the format: gs://<bucket>/<object>.
• https schemed URLs in the format https://<bucket>.storage.googleapis.com/<object>.
• https schemed URLs in the format https://storage.googleapis.com/<bucket>/<object>.

Note that HMAC authentication is used for Google Cloud Storage access.

If authentication is required, the following environment variables may be set when invoking cloud-copy:

• GOOGLE_HMAC_ACCESS_KEY - the HMAC access key to use for authentication.
• GOOGLE_HMAC_SECRET - the HMAC secret to use for authentication.

🚀 Getting Started

Installing Rust

Install a Rust toolchain via rustup.

Installing cloud-copy

To install cloud-copy, run the following command:

cargo install --git https://github.com/stjude-rust-labs/cloud-copy.git

Using cloud-copy

Uploading Files

A file or directory may be transferred to cloud storage by invoking cloud-copy:

$ cloud-copy $SRC $DEST

Where $SRC is the path to the local file or directory to upload and $DEST is a supported cloud storage service URL.

Downloading Files

A file or directory may be transferred from cloud storage by invoking cloud-copy:

$ cloud-copy $SRC $DEST

Where $SRC is a supported cloud storage service URL and $DEST is the local path to copy the file or directory to.

🧠 Running Automated Tests

Automated tests rely on having the following cloud service emulators installed:

• Azurite for Azure Blob Storage.
• Localstack for AWS S3.

Use the following command to run Azurite in a Docker container:

docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite -l /data --blobHost 0.0.0.0 --loose

Use the following command to run Localstack in a Docker container:

docker run -p 4566:4566 localstack/localstack:s3-latest

The tests expect a container/bucket with the name cloud-copy-test to be present.

To create the container with Azurite, use the Azure CLI:

az storage container create --name cloud-copy-test --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1"

To create the bucket with Localstack, use the awslocal tool:

awslocal s3api create-bucket --bucket cloud-copy-test

Finally, run the tests:

cargo test --all

Note: the Azure tests expect *.blob.core.windows.net.localhost to resolve to 127.0.0.1.

✅ Submitting Pull Requests

Before submitting any pull requests, please make sure the code passes the following checks (from the root directory).

# Run the project's tests.
cargo test --all-features

# Run the tests for the examples.
cargo test --examples --all-features

# Ensure the project doesn't have any linting warnings.
cargo clippy --all-features

# Ensure the project passes `cargo fmt`.
cargo fmt --check

# Ensure the docs build.
cargo doc --all-features

🤝 Contributing

Contributions, issues and feature requests are welcome! Feel free to check issues page.

📝 License

This project is licensed as either Apache 2.0 or MIT at your discretion. Additionally, please see the disclaimer that applies to all crates and command line tools made available by St. Jude Rust Labs.

Copyright © 2024-Present St. Jude Children's Research Hospital.