seerdb

Research-grade LSM storage engine with learned data structures.

Embedded key-value storage integrating learned indexes (ALEX), key-value separation (WiscKey), and workload-aware compaction from recent systems research.

Installation

[dependencies]
seerdb = "0.0.4"

Requires nightly Rust:

rustup override set nightly

Quick Start

use seerdb::{DB, DBOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let db = DB::open(DBOptions::default())?;

    // Basic operations
    db.put(b"key", b"value")?;
    let val = db.get(b"key")?;
    db.delete(b"key")?;

    // Batch writes (atomic)
    let mut batch = db.batch();
    batch.put(b"user:1", b"alice");
    batch.put(b"user:2", b"bob");
    batch.commit()?;

    // Range scan
    for result in db.prefix(b"user:")? {
        let (key, value) = result?;
        println!("{:?} = {:?}", key, value);
    }

    // Point-in-time snapshot
    let snapshot = db.snapshot();
    db.put(b"key", b"new_value")?;
    assert_eq!(snapshot.get(b"key")?, val); // snapshot sees old state

    Ok(())
}

Features

• Learned indexes (ALEX) for adaptive key distribution
• Key-value separation (WiscKey) for reduced write amplification
• Merge operators for atomic read-modify-write (counters, lists)
• OCC transactions with snapshot isolation
• Point-in-time snapshots
• Range queries and prefix scans
• Tiered storage with S3/GCS/Azure support
• Compression (ZSTD/LZ4) and SIMD optimizations

Architecture

seerdb is a 7-level LSM tree with:

• Memtable: Partitioned skip list (16 partitions) with lock-free reads via ArcSwap
• WAL: Write-ahead log with configurable sync policies
• SSTable: Two-level index with ALEX learned index for data blocks, bloom filters, LZ4/ZSTD compression
• VLog: Optional WiscKey-style value log for large values (reduces write amplification)
• Compaction: Dostoevsky-inspired tiered/leveled hybrid with lazy leveling

Benchmarks

Run benchmarks with:

cargo bench                              # All benchmarks
cargo bench --bench ycsb_benchmark       # YCSB workloads
cargo bench --bench mixed_workload       # Read/write mix

Testing

cargo test --lib                         # Unit tests (213 tests)
cargo test                               # All tests including integration
cargo test --features failpoints         # Crash/recovery tests

License

Apache License 2.0