NGDB - High-Performance RocksDB Wrapper

A thread-safe, synchronous RocksDB wrapper for Rust that provides a clean, type-safe API with zero async overhead. NGDB abstracts away RocksDB complexity, offering ACID transactions, automatic backups, and distributed replication infrastructure.

Features

• 🚀 Zero Async Overhead - Pure synchronous API leveraging RocksDB's native performance
• 🔒 ACID Transactions - Full transaction support with commit, rollback, and isolation
• 🔐 Type-Safe Collections - Generic key-value storage with compile-time type checking
• 📦 Column Families - Organize multiple typed collections in a single database
• ⚡ Thread-Safe - Safe concurrent access using RocksDB's multi-threaded column families
• 💾 Backup & Restore - Built-in point-in-time backup and disaster recovery
• 🌐 Replication Infrastructure - Framework for distributed replication (network layer not included)
• 🔗 Reference System - Ref<T> type for efficient object relationships
• 🎯 Zero RocksDB Exposure - Complete abstraction—users never interact with RocksDB types
• 📊 Rich Operations - Batch writes, snapshots, multi-get, and functional iteration

Installation

Add to your Cargo.toml:

[dependencies]
ngdb = "2.0.0"
borsh = { version = "1.5", features = ["derive"] }

Quick Start

use ngdb::{DatabaseConfig, Storable, ngdb};

#[ngdb("users")]
struct User {
    id: u64,
    name: String,
    email: String,
}

impl Storable for User {
    type Key = u64;
    fn key(&self) -> Self::Key {
        self.id
    }
}

fn main() -> Result<(), ngdb::Error> {
    let db = DatabaseConfig::new("./data")
        .create_if_missing(true)
        .add_column_family("users")
        .open()?;

    let user = User {
        id: 1,
        name: "Alice".to_string(),
        email: "alice@example.com".to_string(),
    };
    user.save(&db)?;

    let users = User::collection(&db)?;
    if let Some(user) = users.get(&1)? {
        println!("Found: {} <{}>", user.name, user.email);
    }

    Ok(())
}

Core Concepts

Collections

Collections provide type-safe access to column families:

// Save using the generated save() method
product.save(&db)?;

let products = Product::collection(&db)?;

// Single operations
let item = products.get(&id)?;
products.delete(&id)?;

// Batch operations
let items = products.get_many(&[1, 2, 3])?;

// Iteration
products.iter()?.for_each(|product| {
    println!("{}", product.name);
    true // continue iteration
})?;

Transactions

ACID transactions ensure atomic operations:

let txn = db.transaction()?;
let accounts = txn.collection::<Account>("accounts")?;

let mut alice = accounts.get(&1)?.unwrap();
let mut bob = accounts.get(&2)?.unwrap();

alice.balance -= 100;
bob.balance += 100;

accounts.put(&alice)?;
accounts.put(&bob)?;

txn.commit()?;

Batch Operations

Efficient bulk writes:

let users = User::collection(&db)?;
let mut batch = users.batch();
for i in 0..1000 {
    batch.put(&User { id: i, name: format!("User {}", i), email: format!("user{}@example.com", i) })?;
}
batch.commit()?;

Snapshots

Point-in-time consistent reads:

let users = User::collection(&db)?;
let snapshot = users.snapshot()?;

// Read from snapshot while database continues to change
let user = snapshot.get(&1)?;

Backup & Restore

// Create backup
db.backup("./backups")?;

// List backups
let backups = Database::list_backups("./backups")?;

// Restore from backup
Database::restore_from_backup("./backups", "./restored")?;

References

Efficiently store object relationships with automatic lazy loading:

use ngdb::{Ref, ngdb};

#[ngdb("posts")]
struct Post {
    id: u64,
    title: String,
    author: Ref<User>, // Only stores user ID, auto-resolves on access
}

impl Storable for Post {
    type Key = u64;
    fn key(&self) -> Self::Key { self.id }
}

// The #[ngdb] macro automatically implements Referable
// Retrieve post - no mut needed!
let posts = Post::collection(&db)?;
let post = posts.get(&1)?.unwrap();  // Immutable binding

// First call: auto-resolves from DB and caches
let author = post.author.get(&db)?;
println!("Author: {}", author.name);

// Second call: uses cached value, no DB query
let email = post.author.get(&db)?.email;
println!("Email: {}", email);

// Check if resolved without resolving
if post.author.is_resolved() {
    println!("Author is cached!");
}

Distributed Replication

NGDB provides the infrastructure for building distributed systems with eventual consistency. The library handles operation logging and conflict resolution, but you must implement the network layer (HTTP, gRPC, WebSocket, etc.) between nodes.

Architecture

Outbound Replication - Capture local writes as replication logs:

// Your code creates replication logs for writes
let log = create_replication_log(&operation);

// Send to peers via your network layer
send_to_peers(log).await?;

Inbound Replication - Process logs received from peers:

// In your network handler
async fn handle_replication(log: ReplicationLog) -> Result<()> {
    manager.apply_replication(log)?;
    Ok(())
}

NGDB handles:

• Operation serialization
• Timestamp-based conflict resolution
• Atomic batch replication
• Data integrity verification

You implement:

• Network transport (TCP, HTTP, gRPC, etc.)
• Peer discovery and routing
• Authentication and authorization
• Network failure handling

Examples

The repository includes comprehensive examples:

• basic_usage.rs - CRUD operations
• user_struct.rs - Working with custom structs
• derive_macro.rs - Using the #[ngdb] macro
• nested_refs.rs - Object relationships with Ref<T>
• advanced.rs - Advanced features and patterns
• transactions.rs - ACID transactions with rollback
• thread_safe_transactions.rs - Concurrent access patterns
• backup_restore.rs - Disaster recovery
• replication.rs - Basic replication setup
• replication_full.rs - Complete replication system

Run an example:

cargo run --example basic_usage

Performance

NGDB is designed for performance:

• Synchronous operations - No async runtime overhead
• Zero-copy where possible
• RocksDB optimizations - LZ4 compression, jemalloc (non-Windows)
• Efficient batching - Minimize write amplification
• Multi-threaded - Safe concurrent access to column families

License

Dual-licensed under MIT or Apache-2.0.

Acknowledgments

Built on RocksDB, a high-performance embedded key-value store originally developed at Facebook.

Serialization powered by Borsh, a binary serialization format optimized for speed and consistency.

Contributing

Contributions are welcome! Please ensure:

• Code passes cargo test
• Code is formatted with cargo fmt
• Code passes cargo clippy with no warnings
• New features include tests and documentation
• Follow existing code style