ipfrs-storage

Blockstore implementation for IPFRS - the persistent storage layer.

Overview

ipfrs-storage provides high-performance, Rust-native key-value storage for IPFRS blocks with:

• Pluggable Backends: Sled (default), ParityDB, RocksDB
• Hot/Cold Tiering: Automatic data migration based on access patterns
• Differentiable Storage: Version control for tensor gradients (Git-for-Tensors)
• Zero-Copy Reads: Memory-mapped access for large blocks

Key Features

Multi-Backend Support

• Sled: Embedded, pure Rust, optimized for SSDs
• ParityDB: High-performance, designed for blockchain workloads
• RocksDB: Battle-tested, C++ backend with Rust bindings

Differentiable Blockstore (v0.2.0 / v0.3.0)

• Version Control System for model states (Git for Tensors)
• Track gradient updates as IPLD Merkle DAG
• Time-travel to any historical model state
• Commit/checkout operations for reproducible training
• Branch management for collaborative training
• Provenance tracking for XAI (explainable AI)

Performance Optimization

• Memory-mapped I/O for large tensor blocks
• Bloom filters for fast negative lookups
• LRU caching layer above persistent storage
• Batch write operations for high throughput

Architecture

ipfrs-storage
├── traits/       # BlockStore trait definition
├── backends/     # Backend implementations (Sled, ParityDB)
├── cache/        # In-memory LRU cache layer
└── versioning/   # Gradient tracking & version control

Design Principles

• Backend Agnostic: Easy to swap storage engines
• Performance First: Optimized for both read and write throughput
• Memory Efficient: Suitable for edge devices (2GB RAM)
• Crash Safe: ACID guarantees for critical operations

Usage Example

use ipfrs_storage::{BlockStore, SledBackend};
use ipfrs_core::{Block, Cid};

// Initialize storage
let store = SledBackend::open("/path/to/datastore")?;

// Put block
let block = Block::new(data);
store.put(&block).await?;

// Get block by CID
let retrieved = store.get(&cid).await?;

// Check existence (fast bloom filter)
if store.has(&cid).await? {
    // ...
}

Dependencies

• sled - Default embedded database
• parity-db - Optional high-performance backend
• lru - LRU cache implementation
• tokio - Async runtime

References

• IPFRS v0.1.0 Whitepaper (Storage Architecture)
• IPFRS v0.3.0 Whitepaper (Differentiable Storage)