d-engine-server

Complete Raft server with gRPC and storage - batteries included

What is this?

This crate provides a complete Raft server implementation with gRPC networking, persistent storage, and cluster orchestration. It's the server runtime component of d-engine.

d-engine is a lightweight distributed coordination engine written in Rust, designed for embedding into applications that need strong consistency—the consensus layer for building reliable distributed systems.

When to use this crate

• ✅ Embedding server in a larger Rust application
• ✅ Need programmatic access to server APIs
• ✅ Building custom tooling around d-engine
• ✅ Already have your own client implementation

When NOT to use this crate

• ❌ Most applications → Use d-engine for simpler dependency management
• ❌ Need client + server → Use d-engine with features = ["server", "client"]
• ❌ Quick start preferred → Use d-engine for unified API

Quick Start

Add to your Cargo.toml:

[dependencies]
d-engine-server = "0.2"

Embedded Mode (zero-overhead local client)

use d_engine_server::EmbeddedEngine;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = EmbeddedEngine::start_with("config.toml").await?;
    engine.wait_ready(Duration::from_secs(5)).await?;

    let client = engine.client();
    client.put(b"key".to_vec(), b"value".to_vec()).await?;

    engine.stop().await?;
    Ok(())
}

Standalone Mode (independent server)

use d_engine_server::StandaloneServer;
use tokio::sync::watch;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    std::env::set_var("CONFIG_PATH", "config.toml");
    let (_shutdown_tx, shutdown_rx) = watch::channel(());
    StandaloneServer::run(shutdown_rx).await?;
    Ok(())
}

Features

This crate provides:

• gRPC Server - Production-ready Raft RPC implementation
• Storage Backends - File-based and RocksDB storage
• Cluster Orchestration - Node lifecycle and membership management
• Snapshot Coordination - Automatic log compaction
• Watch API (optional) - Real-time state change notifications
• EmbeddedEngine API - High-level API for embedded mode

Architecture

┌─────────────────────────────────────────┐
│ d-engine-server                         │
│                                         │
│  ┌─────────────────────────────────┐   │
│  │ API Layer                       │   │
│  │  - EmbeddedEngine               │   │
│  │  - StandaloneServer             │   │
│  │  - LocalKvClient                │   │
│  └──────────────┬──────────────────┘   │
│                 │                       │
│  ┌──────────────▼──────────────────┐   │
│  │ Node (Raft orchestration)       │   │
│  │  - Leader election              │   │
│  │  - Log replication              │   │
│  │  - Membership changes           │   │
│  └───────┬──────────────┬──────────┘   │
│          │              │               │
│  ┌───────▼────────┐  ┌──▼────────────┐ │
│  │ StorageEngine  │  │ StateMachine  │ │
│  │ (Raft logs)    │  │ (KV store)    │ │
│  └────────────────┘  └───────────────┘ │
│                                         │
│  ┌─────────────────────────────────┐   │
│  │ gRPC Services                   │   │
│  │  - Client RPC (read/write)      │   │
│  │  - Cluster RPC (membership)     │   │
│  │  - Internal RPC (replication)   │   │
│  └─────────────────────────────────┘   │
└─────────────────────────────────────────┘

Storage Backends

File Storage (Default)

Simple file-based storage for development and small deployments.

use d_engine_server::{FileStorageEngine, FileStateMachine};

let storage = Arc::new(FileStorageEngine::new("./logs")?);
let sm = Arc::new(FileStateMachine::new("./sm").await?);

RocksDB Storage (Production)

High-performance embedded database for production use.

[dependencies]
d-engine-server = { version = "0.2", features = ["rocksdb"] }

use d_engine_server::{RocksDBStorageEngine, RocksDBStateMachine};

let storage = Arc::new(RocksDBStorageEngine::new("./data/logs")?);
let sm = Arc::new(RocksDBStateMachine::new("./data/sm").await?);

Custom Storage

Implement [StateMachine] and [StorageEngine] traits:

use d_engine_server::{StateMachine, StorageEngine};

struct MyStateMachine;
impl StateMachine for MyStateMachine {
    // Apply committed entries to your application state
}

struct MyStorageEngine;
impl StorageEngine for MyStorageEngine {
    // Persist Raft logs and metadata
}

See the Server Guide for detailed implementation instructions.

EmbeddedEngine API

High-level API for embedding d-engine in Rust applications:

use d_engine_server::EmbeddedEngine;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Start embedded engine with config file
    let engine = EmbeddedEngine::start_with("d-engine.toml").await?;

    // Wait for leader election
    let leader = engine.wait_ready(Duration::from_secs(5)).await?;
    println!("Leader elected: {}", leader.leader_id);

    // Get zero-overhead client
    let client = engine.client();
    client.put(b"key".to_vec(), b"value".to_vec()).await?;

    // Graceful shutdown
    engine.stop().await?;
    Ok(())
}

Documentation

For comprehensive guides:

• Customize Storage Engine
• Customize State Machine
• Watch Feature Guide
• Consistency Tuning

Examples

See working examples in the repository:

• Single Node Expansion
• Three Nodes Cluster
• Quick Start Embedded

Related Crates

License

MIT or Apache-2.0