Manifold

A high-performance, ACID-compliant embedded database with column families, write-ahead logging, and WASM support.

Manifold is a fork of redb by Christopher Berner, extended with:

• Column families for concurrent writes to independent databases in a single file
• Write-ahead log (WAL) for fast, durable commits
• WASM support via Origin Private File System (OPFS)
• Production-ready crash recovery with comprehensive error handling

Built on redb's solid foundation of copy-on-write B-trees, Manifold adds the architecture needed for high-concurrency workloads while maintaining full ACID guarantees.

---

Quick Start

Basic Usage

use manifold::{Database, ReadableTable, TableDefinition};

const TABLE: TableDefinition<&str, u64> = TableDefinition::new("my_data");

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let db = Database::create("my_db.manifold")?;
    let write_txn = db.begin_write()?;
    {
        let mut table = write_txn.open_table(TABLE)?;
        table.insert("my_key", &123)?;
    }
    write_txn.commit()?;

    let read_txn = db.begin_read()?;
    let table = read_txn.open_table(TABLE)?;
    assert_eq!(table.get("my_key")?.unwrap().value(), 123);

    Ok(())
}

Column Families Example

use manifold::column_family::ColumnFamilyDatabase;
use manifold::TableDefinition;

const USERS: TableDefinition<u64, &str> = TableDefinition::new("users");
const PRODUCTS: TableDefinition<u64, &str> = TableDefinition::new("products");

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Single file, multiple independent databases
    let db = ColumnFamilyDatabase::builder().open("app.manifold")?;

    // Auto-create column families on first access
    let users_cf = db.column_family_or_create("users")?;
    let products_cf = db.column_family_or_create("products")?;

    // Concurrent writes to different column families
    std::thread::scope(|s| {
        s.spawn(|| {
            let txn = users_cf.begin_write()?;
            let mut table = txn.open_table(USERS)?;
            table.insert(&1, &"alice")?;
            txn.commit()
        });

        s.spawn(|| {
            let txn = products_cf.begin_write()?;
            let mut table = txn.open_table(PRODUCTS)?;
            table.insert(&100, &"laptop")?;
            txn.commit()
        });
    });

    Ok(())
}

---

Features

Core Features (from redb)

• ✅ Zero-copy, thread-safe BTreeMap-based API
• ✅ Fully ACID-compliant transactions with snapshot isolation
• ✅ MVCC support for concurrent readers and writers without blocking
• ✅ Crash-safe by default with automatic recovery
• ✅ Savepoints and rollbacks for transaction control
• ✅ Stable file format with upgrade paths

Manifold Extensions

• 🚀 Column Families: Multiple independent databases in a single file with concurrent write transactions
• ⚡ Write-Ahead Log (WAL): Fast durable commits (~0.5ms vs ~5ms) with group commit batching
• 🌐 WASM Support: Full database functionality in browsers via OPFS (Chrome 102+, Edge 102+)
• 🛡️ Production Error Handling: Comprehensive error messages, troubleshooting guides, and recovery procedures
• 📊 Crash Recovery Testing: Process-based crash injection tests validate WAL replay correctness

---

Performance

Manifold vs Vanilla redb 3.1.0

Concurrent Writes (the killer feature):

Threads	Manifold	redb 2.6	Speedup
2	189K ops/sec	75K ops/sec	2.52x
4	293K ops/sec	82K ops/sec	3.56x
8	426K ops/sec	88K ops/sec	4.80x

Why? Column families enable true parallel writes. Vanilla redb serializes all write transactions.

WAL Performance

Configuration	Throughput	Speedup
WITH WAL	273K ops/sec	1.64x
WITHOUT WAL	166K ops/sec	1.00x

Recovery: ~326K entries/sec (20,000 entries in ~61ms)

Read Performance

• Read-heavy workload: 6.74M ops/sec (8 concurrent readers)
• Mixed workload (50/50 R/W): 4.56M ops/sec (16 threads)
• Sequential reads: 4.58M ops/sec

Comparison vs Other Databases

	manifold	lmdb	rocksdb	sled	fjall	sqlite
bulk load	47912ms	10520ms	39093ms	41817ms	91075ms	55585ms
individual writes	51ms	15753ms	20816ms	11038ms	5269ms	6803ms
batch writes	1782ms	6413ms	1355ms	2138ms	727ms	90617ms
nosync writes	2483ms	591141ms	295ms	463ms	853ms	283065ms
len()	0ms	0ms	1413ms	3277ms	1525ms	66ms
random reads	2653ms	1874ms	4202ms	2090ms	4540ms	10131ms
random reads	3097ms	1723ms	4070ms	2185ms	4519ms	10382ms
random range reads	2229ms	1056ms	5000ms	3310ms	4148ms	19729ms
random range reads	2051ms	1057ms	5753ms	3450ms	4146ms	19330ms
random reads (4 threads)	4319ms	2059ms	9234ms	3280ms	5785ms	39302ms
random reads (8 threads)	3432ms	1207ms	9117ms	2126ms	3329ms	47882ms
random reads (16 threads)	2838ms	1093ms	8066ms	2057ms	3321ms	66441ms
random reads (32 threads)	2866ms	1128ms	8173ms	2077ms	3471ms	78306ms
removals	53881ms	7349ms	20050ms	15326ms	8049ms	56281ms
uncompacted size	4.00 GiB	2.59 GiB	1016.07 MiB	2.14 GiB	1010.61 MiB	1.10 GiB
compacted size	N/A	1.26 GiB	459.17 MiB	N/A	1010.61 MiB	562.31 MiB

Source code for benchmark: lmdb_benchmark.rs. Results collected on macOS.

Column Family Concurrent Write Benchmark:

For a more realistic comparison showcasing column family performance with concurrent writers, see the cf_comparison_benchmark.rs which compares Manifold, RocksDB (with column families), and Fjall (with partitions) under concurrent write workloads.

Note on Benchmarks: These benchmarks test single-database performance. Manifold's primary advantage is concurrent writes via column families (see 4.8x speedup vs vanilla redb above). RocksDB and Fjall also support column families/partitions. For column-family-optimized benchmarks, run cargo run --release --bin cf_comparison_benchmark.

---

Column Families Architecture

What Are Column Families?

Column families allow multiple independent redb databases to coexist in a single physical file. Each column family:

• Has its own B-tree root and transaction isolation
• Supports independent write transactions (no cross-CF locking)
• Maintains ACID guarantees within the column family
• Shares the same file with other column families

When to Use Column Families

✅ Good fit:

• Multi-tenant applications (one CF per tenant)
• Domain separation (users, products, orders as separate CFs)
• Concurrent write workloads where different threads write to different data domains
• Scenarios where you'd otherwise use multiple database files

❌ Not needed:

• Single-threaded write workloads
• Cross-domain transactions (use tables within a single CF instead)
• Simple key-value storage (vanilla redb is simpler)

Architecture Details

┌─────────────────────────────────────────┐
│         app.manifold (single file)      │
├─────────────────────────────────────────┤
│  Master Header (CRC-protected)          │
├─────────────────────────────────────────┤
│  Column Family "users"    (1GB)         │
│  - Independent redb instance            │
│  - Own transaction isolation            │
├─────────────────────────────────────────┤
│  Column Family "products" (512MB)       │
│  - Concurrent writes with "users"       │
│  - Own transaction isolation            │
├─────────────────────────────────────────┤
│  Column Family "orders"   (2GB)         │
│  - Independent write transactions       │
└─────────────────────────────────────────┘

See docs/design.md for implementation details.

---

Write-Ahead Log (WAL)

Why WAL?

Traditional databases fsync the entire B-tree on every commit (~5ms). WAL appends commit records to a sequential log and fsyncs only that (~0.5ms), giving ~10x faster commits.

Performance Characteristics

• Commit latency: ~0.5ms (WAL) vs ~5ms (no WAL)
• Throughput: 250K ops/sec @ 8 threads with WAL enabled
• Group commit: Batches concurrent commits into single fsync
• Recovery: Automatic on database open (~300K entries/sec)

Durability Modes

use manifold::Durability;

let txn = db.begin_write()?;
// ...

// Default: fsync on commit (data survives crashes)
txn.set_durability(Durability::Immediate)?;
txn.commit()?;

// Fast path: no fsync (data may be lost on crash)
txn.set_durability(Durability::None)?;
txn.commit()?;

Recovery Guarantees

• ✅ Committed transactions with Durability::Immediate: Survive crashes
• ⚠️ Committed transactions with Durability::None: May be lost (depends on checkpoint)
• ❌ Uncommitted transactions: Always lost (expected ACID behavior)

See docs/recovery_guarantees.md for detailed recovery semantics.

---

WASM Support

Manifold runs in browsers using the Origin Private File System (OPFS):

// worker.js
import init, { WasmDatabase } from './manifold.js';

await init();
const db = await WasmDatabase.new("app.db");
const cf = db.column_family_or_create("users");

// Write data
cf.write("user_1", "alice");

// Read data
const value = cf.read("user_1");
console.log(value); // "alice"

Requirements:

• Modern browser (Chrome 102+, Edge 102+)
• Web Worker context (OPFS sync access requirement)
• HTTPS or localhost (secure context)

Performance: Same as native (OPFS provides synchronous file access in workers)

See examples/wasm/ for complete examples.

---

Documentation

Guides

• Design Document - Architecture and implementation details
• WAL Design - Write-ahead log implementation
• Recovery Guarantees - Crash recovery and durability semantics
• Troubleshooting Guide - Common errors and solutions

Examples

• Column Families - Multi-tenant and domain separation
• Error Handling - Production error handling patterns
• WAL Comparison - WAL vs no-WAL performance
• WASM Usage - Browser database with OPFS

API Documentation

• Coming soon: docs.rs link once published

---

Error Handling

Manifold provides comprehensive error handling with clear messages:

use manifold::column_family::ColumnFamilyDatabase;

match ColumnFamilyDatabase::open("data.manifold") {
    Ok(db) => { /* use database */ }
    Err(e) => {
        eprintln!("Database error: {}", e);
        // Error includes context: corruption details, I/O errors, etc.
        // See TROUBLESHOOTING.md for recovery procedures
    }
}

Error categories:

• Storage errors (corruption, I/O, disk full)
• Table errors (type mismatches, missing tables)
• Transaction errors (conflicts, isolation violations)
• Column family errors (not found, already exists)

See TROUBLESHOOTING.md for common issues and solutions.

---

Testing

Manifold includes comprehensive test coverage:

• 98 unit tests: Core functionality
• 9 crash recovery tests: Process-based crash injection validates WAL replay
• Multiple benchmark suites: Performance validation

Run tests:

cargo test --lib                      # Unit tests
cargo test --test crash_recovery_tests # Crash injection tests (Unix/macOS)

Run benchmarks:

cargo run --release --bin lmdb_benchmark           # vs other databases
cargo run --release --bin redb_comparison_benchmark # vs vanilla redb
cargo bench                                         # criterion benchmarks

---

Production Readiness

Manifold is production-ready with:

• ✅ Stable file format (inherited from redb)
• ✅ Comprehensive error handling (Phase 2 complete)
• ✅ Crash recovery validation (process-based crash tests)
• ✅ Clear documentation (guides, examples, troubleshooting)
• ✅ Performance benchmarks (validated against redb and competitors)
• ✅ WASM support (full functionality in browsers)

Known limitations:

• Column families are auto-created; manual sizing planned for v3.2
• WAL checkpointing is time/size-based; manual checkpoint API planned
• No replication or distributed features (single-process database)

See FINALIZATION_PLAN.md for roadmap.

---

Credits

Manifold is a fork of redb by Christopher Berner.

Original redb features:

• Copy-on-write B-tree implementation
• ACID transactions with MVCC
• Zero-copy reads
• Savepoints and recovery

Manifold additions:

• Column family architecture by Tom (Hyperspatial)
• Write-ahead log implementation
• WASM/OPFS backend
• Production error handling and crash recovery

We're deeply grateful to Christopher Berner for creating redb and maintaining it as a high-quality, production-ready embedded database. Manifold builds on that solid foundation.

---

License

Licensed under either of:

• Apache License, Version 2.0
• MIT License

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

---

Development

Building from Source

git clone https://github.com/hyperbasedev/manifold
cd manifold
cargo build --release

Running Tests

cargo test --all-targets
cargo clippy --all-targets

Extra Development Dependencies

For benchmarking and fuzzing:

cargo install cargo-deny --locked
cargo install cargo-fuzz --locked

For WASM development:

cargo install wasm-pack

---

Status: Stable and actively maintained. Phase 2 (Production Error Handling) complete. Phase 3 (API Polish) in progress.