excelstream

🦀 High-performance streaming Excel, CSV & Parquet library for Rust with constant memory usage

✨ Highlights

• 📊 XLSX, CSV & Parquet Support - Read/write Excel, CSV, and Parquet files
• 📉 Constant Memory - ~3-35 MB regardless of file size
• ☁️ Cloud Streaming - Direct S3/GCS uploads with ZERO temp files
• ⚡ High Performance - 94K rows/sec (S3), 1.2M rows/sec (CSV)
• 🔄 True Streaming - Process files row-by-row, no buffering
• 🗜️ Parquet Conversion - Stream Excel ↔ Parquet with constant memory
• 🐳 Production Ready - Works in 256 MB containers

🔥 What's New in v0.17.0

Multi-Cloud Explicit Credentials - Upload to multiple S3-compatible clouds with different credentials!

use excelstream::cloud::{S3ExcelWriter, S3ExcelReader};
use s_zip::cloud::S3ZipWriter;
use aws_sdk_s3::{Client, config::Credentials};

// Create clients with different credentials for each cloud
let aws_client = Client::from_conf(aws_config);
let minio_client = Client::from_conf(minio_config);

// Upload to multiple clouds simultaneously
let s3_writer = S3ZipWriter::new(aws_client, "bucket", "file.xlsx").await?;
let mut writer = S3ExcelWriter::from_s3_writer(s3_writer);
writer.write_header_bold(["Name", "Value"]).await?;
writer.save().await?;

Features:

• 🔑 Explicit credentials - No environment variables needed
• 🌍 Multi-cloud support - AWS, MinIO, R2, Spaces, B2
• 🚀 True streaming - Only 19-20 MB memory for 100K rows
• ⚡ Concurrent uploads - Upload to multiple clouds simultaneously

Also includes: Parquet support from v0.16.0 (Excel ↔ Parquet streaming)

See full changelog | Multi-cloud guide →

---

📦 Quick Start

Installation

[dependencies]
excelstream = "0.17"

# Optional features
excelstream = { version = "0.17", features = ["cloud-s3"] }        # S3 support
excelstream = { version = "0.17", features = ["cloud-gcs"] }       # GCS support
excelstream = { version = "0.17", features = ["parquet-support"] } # Parquet conversion

Write Excel (Local)

use excelstream::ExcelWriter;

let mut writer = ExcelWriter::new("output.xlsx")?;

// Write 1M rows with only 3 MB memory!
writer.write_header_bold(&["ID", "Name", "Amount"])?;
for i in 1..=1_000_000 {
    writer.write_row(&[&i.to_string(), "Item", "1000"])?;
}
writer.save()?;

Read Excel (Streaming)

use excelstream::ExcelReader;

let mut reader = ExcelReader::open("large.xlsx")?;

// Process 1 GB file with only 12 MB memory!
for row in reader.rows("Sheet1")? {
    let row = row?;
    println!("{:?}", row.to_strings());
}

S3 Streaming (v0.14+)

use excelstream::cloud::S3ExcelWriter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut writer = S3ExcelWriter::builder()
        .bucket("reports")
        .key("sales.xlsx")
        .build()
        .await?;

    writer.write_header_bold(["Date", "Revenue"]).await?;
    writer.write_row(["2024-01-01", "125000"]).await?;
    writer.save().await?;  // Streams to S3, no disk!
    Ok(())
}

More examples →

---

🎯 Why ExcelStream?

The Problem: Traditional libraries load entire files into memory

// ❌ Traditional: 1 GB file = 1+ GB RAM (OOM in containers!)
let workbook = Workbook::new("huge.xlsx")?;

The Solution: True streaming with constant memory

// ✅ ExcelStream: 1 GB file = 12 MB RAM
let mut reader = ExcelReader::open("huge.xlsx")?;
for row in reader.rows("Sheet1")? { /* streaming! */ }

Performance Comparison

Operation	Traditional	ExcelStream	Improvement
Write 1M rows	100+ MB	2.7 MB	97% less memory
Read 1GB file	❌ Crash	12 MB	Works!
S3 upload 500K rows	Temp file	34 MB	Zero disk
K8s pod (256MB)	❌ OOMKilled	✅ Works	Production ready

---

☁️ Cloud Features

S3 Direct Streaming (v0.14)

Upload Excel files directly to S3 with ZERO temp files:

cargo add excelstream --features cloud-s3

Performance (Real AWS S3):

Dataset	Memory	Throughput	Temp Files
10K rows	15 MB	11K rows/s	ZERO ✅
100K rows	23 MB	45K rows/s	ZERO ✅
500K rows	34 MB	94K rows/s	ZERO ✅

Perfect for:

• ✅ AWS Lambda (read-only filesystem)
• ✅ Docker containers (no disk space)
• ✅ Kubernetes CronJobs (limited memory)

See S3 performance details →

S3-Compatible Services (v0.17+)

Stream to AWS S3, MinIO, Cloudflare R2, DigitalOcean Spaces, and other S3-compatible services with explicit credentials - no environment variables needed!

use excelstream::cloud::{S3ExcelWriter, S3ExcelReader};
use s_zip::cloud::S3ZipWriter;
use aws_sdk_s3::{Client, config::Credentials};

// Example 1: AWS S3 with explicit credentials
let aws_creds = Credentials::new(
    "AKIA...",           // access_key_id
    "secret...",         // secret_access_key
    None, None, "aws"
);
let aws_config = aws_sdk_s3::Config::builder()
    .credentials_provider(aws_creds)
    .region(aws_sdk_s3::config::Region::new("ap-southeast-1"))
    .build();
let aws_client = Client::from_conf(aws_config);

// Example 2: MinIO with explicit credentials
let minio_creds = Credentials::new("minioadmin", "minioadmin", None, None, "minio");
let minio_config = aws_sdk_s3::Config::builder()
    .credentials_provider(minio_creds)
    .endpoint_url("http://localhost:9000")
    .region(aws_sdk_s3::config::Region::new("us-east-1"))
    .force_path_style(true)  // Required for MinIO
    .build();
let minio_client = Client::from_conf(minio_config);

// Example 3: Cloudflare R2 with explicit credentials
let r2_creds = Credentials::new("access_key", "secret_key", None, None, "r2");
let r2_config = aws_sdk_s3::Config::builder()
    .credentials_provider(r2_creds)
    .endpoint_url("https://<account-id>.r2.cloudflarestorage.com")
    .region(aws_sdk_s3::config::Region::new("auto"))
    .build();
let r2_client = Client::from_conf(r2_config);

// Write Excel file to ANY S3-compatible service
let s3_writer = S3ZipWriter::new(aws_client.clone(), "my-bucket", "report.xlsx").await?;
let mut writer = S3ExcelWriter::from_s3_writer(s3_writer);
writer.write_header_bold(["Name", "Value"]).await?;
writer.write_row(["Test", "123"]).await?;
writer.save().await?;

// Read Excel file from ANY S3-compatible service
let mut reader = S3ExcelReader::from_s3_client(aws_client, "my-bucket", "data.xlsx").await?;
for row in reader.rows("Sheet1")? {
    println!("{:?}", row?.to_strings());
}

Supported Services:

Service	Endpoint Example	Region
AWS S3	(default)	us-east-1, ap-southeast-1, etc.
MinIO	http://localhost:9000	us-east-1
Cloudflare R2	https://<account>.r2.cloudflarestorage.com	auto
DigitalOcean Spaces	https://nyc3.digitaloceanspaces.com	us-east-1
Backblaze B2	https://s3.us-west-000.backblazeb2.com	us-west-000
Linode	https://us-east-1.linodeobjects.com	us-east-1

✨ Key Features:

• 🔑 Explicit credentials - no environment variables needed
• 🌍 Multi-cloud support - use different credentials for each cloud
• 🚀 True streaming - only 19-20 MB memory for 100K rows
• ⚡ Concurrent uploads - upload to multiple clouds simultaneously
• 🔒 Type-safe - full compile-time checking

🔑 Full Multi-Cloud Guide → MULTI_CLOUD_CONFIG.md - Complete examples for AWS, MinIO, R2, Spaces, and B2!

GCS Direct Streaming (v0.14)

Upload Excel files directly to Google Cloud Storage with ZERO temp files:

cargo add excelstream --features cloud-gcs

use excelstream::cloud::GCSExcelWriter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut writer = GCSExcelWriter::builder()
        .bucket("my-bucket")
        .object("report.xlsx")
        .build()
        .await?;

    writer.write_header_bold(["Month", "Sales"]).await?;
    writer.write_row(["January", "50000"]).await?;
    writer.save().await?; // ✅ Streams directly to GCS!
    Ok(())
}

Perfect for:

• ✅ Cloud Run (read-only filesystem)
• ✅ Cloud Functions (no disk space)
• ✅ GKE workloads (limited memory)

See GCS example →

HTTP Streaming

Stream Excel files directly to web responses:

use excelstream::cloud::HttpExcelWriter;

async fn download() -> impl IntoResponse {
    let mut writer = HttpExcelWriter::new();
    writer.write_row(&["Data"])?;
    ([(header::CONTENT_TYPE, "application/vnd....")], writer.finish()?)
}

HTTP streaming guide →

---

📊 CSV Support

13.5x faster than Excel for CSV workloads:

use excelstream::csv::CsvWriter;

let mut writer = CsvWriter::new("data.csv")?;
writer.write_row(&["A", "B", "C"])?;  // 1.2M rows/sec!
writer.save()?;

Features:

• ✅ Zstd compression (.csv.zst - 2.9x smaller)
• ✅ Auto-detection (.csv, .csv.gz, .csv.zst)
• ✅ Streaming (< 5 MB memory)

CSV examples →

---

🗜️ Parquet Support (v0.16+)

Convert between Excel and Parquet with constant memory streaming:

cargo add excelstream --features parquet-support

Excel → Parquet

use excelstream::parquet::ExcelToParquetConverter;

let converter = ExcelToParquetConverter::new("data.xlsx")?;
let rows = converter.convert_to_parquet("output.parquet")?;
println!("Converted {} rows", rows);

Parquet → Excel

use excelstream::parquet::ParquetToExcelConverter;

let converter = ParquetToExcelConverter::new("data.parquet")?;
let rows = converter.convert_to_excel("output.xlsx")?;
println!("Converted {} rows", rows);

Streaming with Progress

let converter = ParquetToExcelConverter::new("large.parquet")?;
converter.convert_with_progress("output.xlsx", |current, total| {
    println!("Progress: {}/{} rows", current, total);
})?;

Features:

• ✅ Constant memory - Processes in 10K row batches
• ✅ All data types - Strings, numbers, booleans, dates, timestamps
• ✅ Progress tracking - Monitor large conversions
• ✅ High performance - Efficient columnar format handling

Use Cases:

• Convert Excel reports to Parquet for data lakes
• Export Parquet data to Excel for analysis
• Integrate with Apache Arrow/Spark workflows

Parquet examples →

---

🚀 Use Cases

1. Large File Processing

// Process 500 MB Excel with only 25 MB RAM
let mut reader = ExcelReader::open("customers.xlsx")?;
for row in reader.rows("Sales")? {
    // Process row-by-row, constant memory!
}

2. Database Exports

// Export 1M database rows to Excel
let mut writer = ExcelWriter::new("export.xlsx")?;
let rows = db.query("SELECT * FROM large_table")?;
for row in rows {
    writer.write_row(&[row.get(0), row.get(1)])?;
}
writer.save()?;  // Only 3 MB memory used!

3. Cloud Pipelines

// Lambda function: DB → Excel → S3
let mut writer = S3ExcelWriter::builder()
    .bucket("data-lake").key("export.xlsx").build().await?;

let rows = db.query_stream("SELECT * FROM events").await?;
while let Some(row) = rows.next().await {
    writer.write_row(row).await?;
}
writer.save().await?;  // No temp files, no disk!

---

📚 Documentation

• API Docs - Full API reference
• Examples - Code examples for all features
• CHANGELOG - Version history
• Performance - Detailed benchmarks
• Multi-Cloud Config - AWS, MinIO, R2, Spaces, B2 setup

Key Topics

• Excel Writing - Basic & advanced writing
• Excel Reading - Streaming read
• S3 Streaming - AWS S3 uploads
• Multi-Cloud Config - Multi-cloud credentials
• GCS Streaming - Google Cloud Storage uploads
• CSV Support - CSV operations
• Parquet Conversion - Excel ↔ Parquet
• Styling - Cell formatting & colors

---

🔧 Features

Feature	Description
default	Core Excel/CSV with Zstd compression
cloud-s3	S3 direct streaming (async)
cloud-gcs	GCS direct streaming (async)
cloud-http	HTTP response streaming
parquet-support	Parquet ↔ Excel conversion
serde	Serde serialization support
parallel	Parallel processing with Rayon

---

⚡ Performance

Memory Usage (Constant):

• Excel write: 2.7 MB (any size)
• Excel read: 10-12 MB (any size)
• S3 streaming: 30-35 MB (any size)
• CSV write: < 5 MB (any size)

Throughput:

• Excel write: 42K rows/sec
• Excel read: 50K rows/sec
• S3 streaming: 94K rows/sec
• CSV write: 1.2M rows/sec

---

🛠️ Migration from v0.13

S3ExcelWriter is now async:

// OLD (v0.13 - sync)
writer.write_row(&["a", "b"])?;

// NEW (v0.14 - async)
writer.write_row(["a", "b"]).await?;

All other APIs unchanged!

---

📋 Requirements

• Rust 1.70+
• Optional: AWS credentials for S3 features

---

🤝 Contributing

Contributions welcome! Please read CONTRIBUTING.md.

---

📄 License

MIT License - See LICENSE for details

---

🙏 Credits

• Built with s-zip for streaming ZIP
• AWS SDK for Rust
• All contributors and users!

---

Need help? Open an issue | Questions? Discussions