ruv-swarm-persistence 🗄️

High-performance, ACID-compliant persistence layer for RUV-Swarm with SQLite and cross-platform support.

🎯 Overview

ruv-swarm-persistence is the foundational data persistence layer for the RUV-Swarm distributed agent system. It provides robust, type-safe data storage with full ACID compliance, real-time persistence, and seamless cross-platform compatibility.

Built with modern Rust principles, this crate delivers memory-safe database operations, connection pooling, transaction support, and a flexible repository pattern for managing swarm agent state, task coordination, event sourcing, and performance metrics.

✨ Key Features

🔒 ACID Compliance & Reliability

• Full ACID transaction support with automatic rollback
• WAL (Write-Ahead Logging) mode for optimal concurrency
• Automatic connection pooling with configurable limits
• Built-in data integrity checks and foreign key constraints

🚀 High Performance

• Connection pooling with R2D2 for efficient resource management
• Optimized SQLite configuration with NORMAL synchronous mode
• Comprehensive indexing strategy for fast queries
• Automatic vacuum and checkpoint operations

🌐 Cross-Platform Support

• Native: SQLite backend for desktop and server deployments
• WASM: IndexedDB backend for browser-based applications
• Memory: In-memory storage for testing and development
• Unified async/await API across all platforms

🏗️ Developer Experience

• Type-safe query builder with compile-time validation
• Repository pattern for clean data access abstractions
• Comprehensive error handling with detailed error types
• Extensive test coverage with property-based testing

📊 Rich Data Models

• Agents: Complete lifecycle management with heartbeat tracking
• Tasks: Priority-based task queue with dependency resolution
• Events: Event sourcing with timestamp and sequence tracking
• Messages: Inter-agent communication with read receipts
• Metrics: Performance monitoring with aggregation support

🚀 Installation

Add to your Cargo.toml:

[dependencies]
ruv-swarm-persistence = "0.1.0"

# For WASM support
[target.'cfg(target_arch = "wasm32")'.dependencies]
ruv-swarm-persistence = { version = "0.1.0", features = ["wasm"] }

📖 Quick Start

Basic Usage

use ruv_swarm_persistence::{init_storage, AgentModel, TaskModel, Storage};
use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize storage (SQLite on native, IndexedDB on WASM)
    let storage = init_storage(Some("swarm.db")).await?;
    
    // Create and store an agent
    let mut agent = AgentModel::new(
        "worker-001".to_string(),
        "researcher".to_string(),
        vec!["analysis".to_string(), "data-processing".to_string()]
    );
    
    storage.store_agent(&agent).await?;
    
    // Retrieve agent
    let retrieved = storage.get_agent(&agent.id).await?;
    println!("Agent retrieved: {:?}", retrieved);
    
    Ok(())
}

Transaction Support

use ruv_swarm_persistence::{SqliteStorage, Storage, TaskModel, TaskPriority};

async fn atomic_task_processing() -> Result<(), Box<dyn std::error::Error>> {
    let storage = SqliteStorage::new("swarm.db").await?;
    
    // Begin transaction
    let mut tx = storage.begin_transaction().await?;
    
    // Multiple operations in single transaction
    let task1 = TaskModel::new(
        "analysis".to_string(),
        serde_json::json!({"dataset": "sales_data.csv"}),
        TaskPriority::High
    );
    
    let task2 = TaskModel::new(
        "report".to_string(), 
        serde_json::json!({"template": "quarterly"}),
        TaskPriority::Medium
    );
    
    // Both tasks stored atomically
    storage.store_task(&task1).await?;
    storage.store_task(&task2).await?;
    
    // Commit all changes
    tx.commit().await?;
    
    Ok(())
}

Query Builder

use ruv_swarm_persistence::{QueryBuilder, AgentModel};

// Type-safe query construction
let query = QueryBuilder::<AgentModel>::new("agents")
    .where_eq("status", "active")
    .where_like("agent_type", "research%")
    .order_by("created_at", true)
    .limit(50)
    .build();

println!("Generated SQL: {}", query);
// Output: SELECT * FROM agents WHERE status = 'active' AND agent_type LIKE 'research%' ORDER BY created_at DESC LIMIT 50

Event Sourcing

use ruv_swarm_persistence::{EventModel, Storage};

async fn track_agent_events(storage: &dyn Storage) -> Result<(), Box<dyn std::error::Error>> {
    let event = EventModel::new(
        "agent_started".to_string(),
        Some("agent-123".to_string()),
        serde_json::json!({
            "startup_time": "2024-01-01T10:00:00Z",
            "capabilities": ["nlp", "analysis"]
        })
    );
    
    storage.store_event(&event).await?;
    
    // Query recent events
    let recent_events = storage.get_events_since(
        chrono::Utc::now().timestamp() - 3600 // Last hour
    ).await?;
    
    println!("Recent events: {}", recent_events.len());
    Ok(())
}

Real-time Metrics

use ruv_swarm_persistence::{MetricModel, Storage};

async fn performance_monitoring(storage: &dyn Storage) -> Result<(), Box<dyn std::error::Error>> {
    // Store performance metrics
    let cpu_metric = MetricModel::new(
        "cpu_usage".to_string(),
        Some("agent-456".to_string()),
        85.5,
        "percent".to_string(),
        [("host".to_string(), "worker-node-01".into())].into()
    );
    
    storage.store_metric(&cpu_metric).await?;
    
    // Query aggregated metrics
    let metrics = storage.get_aggregated_metrics(
        "cpu_usage",
        chrono::Utc::now().timestamp() - 86400, // Last 24 hours
        chrono::Utc::now().timestamp()
    ).await?;
    
    println!("CPU metrics over 24h: {}", metrics.len());
    Ok(())
}

🏗️ Database Schema

The persistence layer uses a comprehensive schema optimized for swarm operations:

Core Tables

• agents - Agent lifecycle and metadata management
• tasks - Priority-based task queue with dependencies
• events - Event sourcing for audit trails
• messages - Inter-agent communication
• metrics - Performance and monitoring data
• schema_migrations - Version tracking for database evolution

Optimized Indexing

-- Performance-optimized indexes
CREATE INDEX idx_agents_status ON agents(status);
CREATE INDEX idx_tasks_priority ON tasks(priority DESC);  
CREATE INDEX idx_events_timestamp ON events(timestamp);
CREATE INDEX idx_messages_unread ON messages(to_agent, read);
CREATE INDEX idx_metrics_type_time ON metrics(metric_type, timestamp);

PRAGMA Configuration

-- Optimized for concurrent access
PRAGMA foreign_keys = ON;
PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;

🎛️ Configuration

Connection Pool Settings

use ruv_swarm_persistence::SqliteStorage;

// Custom pool configuration
let storage = SqliteStorage::builder()
    .max_connections(32)
    .min_idle_connections(4)
    .connection_timeout(Duration::from_secs(60))
    .idle_timeout(Duration::from_secs(600))
    .build("swarm.db")
    .await?;

Feature Flags

[dependencies]
ruv-swarm-persistence = { 
    version = "0.1.0", 
    features = ["wasm", "migrations"] 
}

Available features:

• wasm - Enable WASM/IndexedDB support
• migrations - Database migration utilities
• rusqlite-backend (default) - Native SQLite support

📊 Performance Benchmarks

Benchmarks on Apple M1 Pro with SQLite WAL mode:

Run cargo bench for platform-specific benchmarks

🧪 Testing

The crate includes comprehensive test coverage:

# Run all tests
cargo test

# Run with coverage
cargo tarpaulin --out html

# Property-based testing
cargo test --features proptest

# WASM testing  
wasm-pack test --node

🔧 Advanced Usage

Custom Storage Implementation

use ruv_swarm_persistence::{Storage, StorageError, AgentModel};
use async_trait::async_trait;

pub struct CustomStorage {
    // Your implementation
}

#[async_trait]
impl Storage for CustomStorage {
    type Error = StorageError;
    
    async fn store_agent(&self, agent: &AgentModel) -> Result<(), Self::Error> {
        // Custom storage logic
        Ok(())
    }
    
    // Implement other required methods...
}

Migration Management

use ruv_swarm_persistence::migrations::{Migration, MigrationRunner};

let migration = Migration::new(
    2,
    "add_agent_tags",
    "ALTER TABLE agents ADD COLUMN tags TEXT DEFAULT '{}';"
);

let runner = MigrationRunner::new(&storage);
runner.run_migration(migration).await?;

📚 API Reference

Core Traits

• Storage - Main persistence interface
• Transaction - ACID transaction support
• Repository<T> - Type-safe data access pattern

Data Models

• AgentModel - Swarm agent representation
• TaskModel - Work item with priority and dependencies
• EventModel - Event sourcing record
• MessageModel - Inter-agent communication
• MetricModel - Performance and monitoring data

Error Types

• StorageError - Comprehensive error handling
• Database - SQL execution errors
• Serialization - JSON serialization issues
• NotFound - Resource lookup failures
• Transaction - ACID transaction errors

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

git clone https://github.com/ruvnet/ruv-fann.git
cd ruv-fann/ruv-swarm/crates/ruv-swarm-persistence
cargo build
cargo test

📋 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Related Projects

• ruv-fann - Main repository and neural network foundation
• ruv-swarm - Distributed agent coordination system
• ruv-swarm-core - Core swarm coordination logic

📞 Support

• Documentation: docs.rs/ruv-swarm-persistence
• Issues: GitHub Issues
• Discussions: GitHub Discussions

Created by rUv - Building the future of distributed AI systems with Rust 🦀