sulfite
| Crates.io | sulfite |
| lib.rs | sulfite |
| version | 0.0.3 |
| created_at | 2024-11-06 19:27:03.16568+00 |
| updated_at | 2024-11-21 14:52:39.231799+00 |
| description | A high-level S3 client built on AWS SDK for Rust for bandwidth saturation |
| homepage | https://github.com/clearviewai/sulfite |
| repository | https://github.com/clearviewai/sulfite |
| max_upload_size | |
| id | 1438707 |
| size | 47,418 |
Terence Z. Liu (terencezl)
documentation
README
sulfite
Overview
sulfite is a high-level S3 client built on AWS SDK for Rust for even better ease of use, reliability, and bandwidth saturation (>50 Gbps).
The name: SO3^2-, an anion, implying a companion to some other cation (application), is commonly used as a preservative in wines and dried fruits (preserve to S3). It's S3 with an O in the middle, a play on oxidization.
Motivation
The AWS SDK is a little low-level for users to take advantage of the concurrency & parallelism, with the following challenges:
- You need to orchestrate the parallel multipart download & upload for large files.
- The built-in retry settings are too low-level, and we allow installing higher-level retries.
- The async API doesn't agree well with the filesystem, especially for streaming small chunks from/to disk.
To address them, we provide implementations for the parallel multipart download & upload, and higher-level retries. We also make sure the on-disk file is adequately buffered to avoid task-threading overhead.
License
This project is licensed under the MIT license.