siftdb-core

Core engine for SiftDB: a grep-native, agent-oriented database for code and text collections.

Features

• Append-only storage with CRCs and frame format
• Line table generation with delta-varints
• Path and handle mapping
• O(1) inverted index for instant term lookups
• Memory-mapped FST indexes for efficient term storage
• Substring and regex search (via trigram index in earlier milestones)

Quick Start

cargo add siftdb-core

Example

use siftdb_core::SiftDB;

fn main() -> anyhow::Result<()> {
    let db = SiftDB::open("project.sift")?;
    let mut snapshot = db.snapshot()?;
    let hits = snapshot.find("async", Some("**/*.rs"), Some(10))?;
    for h in hits { println!("{}:{}: {}", h.path, h.line, h.text);}    
    Ok(())
}

Performance

• 24M+ queries/sec for non-existent terms (pure O(1) lookup)
• Performance scales with number of matching files (O(k) processing)

See the root benchmarks/README.md for full methodology and results.

License

MIT