🦀 minikv

A distributed, multi-tenant key-value & object store written in Rust

minikv provides strong consistency (Raft + 2PC), durability (WAL), and production-grade observability, security, and multi-tenancy — all in a modern Rust codebase.

Built in public as a learning-by-doing project — now evolved into a complete, reference implementation of distributed systems in Rust.

🚦 What's New in v0.7.0

minikv v0.7.0 brings advanced data management and query capabilities :

• Secondary indexes : Search keys by value content with GET /search?value=<substring>
• Multi-key transactions : Execute multiple operations atomically with POST /transaction
• Streaming/batch import/export : Bulk data operations with POST /admin/import & GET /admin/export
• Durable S3-backed object store : Persistent storage for S3-compatible API via pluggable backends

Previous highlights (v0.6.0) : enterprise security, multi-tenancy, encryption at rest, quotas, audit logging, persistent backends, watch/subscribe system.

📚 Table of Contents

• What is minikv?
• Tech Stack
• Quick Start
• Architecture
• Performance
• Features
• Roadmap
• Story
• Documentation
• Development
• Contributing
• Contact

🤔 What is minikv?

minikv is a distributed key-value store written in Rust, designed for simplicity, speed, and reliability.

Who is this for ?
minikv is for engineers learning distributed systems, teams experimenting with Rust-based infrastructure, and anyone curious about consensus, durability, and system trade-offs.

• Clustered : Raft consensus and 2PC for transactional writes
• Virtual Sharding : 256 vshards for elastic scaling & balancing
• WAL : Write-ahead log for durability
• gRPC for node communication, HTTP REST & S3 API for clients
• Bloom filters, snapshots, watch/subscribe for performance & reactivity

🛠 Tech Stack

• Rust – core logic
• Shell – orchestration/automation
• JavaScript – benchmarks, tools
• Makefile – build flows

⚡ Quick Start

git clone https://github.com/whispem/minikv.git
cd minikv
cargo build --release

# Start a node
cargo run -- --config config.example.toml

# API examples
curl localhost:8080/health/ready   # readiness
curl localhost:8080/metrics        # Prometheus metrics
curl localhost:8080/admin/status   # admin dashboard

# Create API key (admin)
curl -X POST http://localhost:8080/admin/keys -d '{"role":"ReadWrite","tenant_id":"acme"}'

# S3 (demo)
curl -X PUT localhost:8080/s3/mybucket/mykey -d 'hello minikv!'
curl localhost:8080/s3/mybucket/mykey

For cluster setup and advanced options, see the documentation.

📐 Architecture

• Raft: consensus and leader election
• 2PC: atomic distributed/batch writes
• Virtual Shards: scale and rebalance across 256 partitions
• Pluggable Storage: in-memory, RocksDB, Sled
• Admin API: HTTP endpoints for status, metrics and config
• Config: via environment, file or CLI flags

🚀 Performance

• Write throughput : over 50,000 operations/sec (single node, in-memory)
• Sub-millisecond read latency
• Cluster tested (3–5 nodes, commodity VMs)
• Built-in Prometheus metrics

🌟 Features

Distributed Core

• Raft consensus (multi-node, strong consistency)
• Two-phase commit (2PC) for atomic multi-key transactions
• 256 virtual shards for cluster scaling and rebalancing
• Write-ahead log (WAL) for durability
• Auto-rebalancing, graceful leader failover, hot-join and node removal

Data Management

• Time-To-Live keys (TTL)
• LZ4 compression (configurable)
• Bloom filters and index snapshots
• Pluggable and persistent storage: in-memory, RocksDB, Sled
• Batch & range operations, prefix queries

API

• HTTP REST (CRUD, batch, range, admin)
• S3-compatible API (with TTL extensions)
• gRPC (internal)
• WebSocket and SSE endpoints for real-time watch/subscribe events

Security & Multi-tenancy

• API keys (Argon2) and JWT authentication
• Role-based access control (RBAC) and audit logging
• Multi-tenant isolation
• AES-256-GCM encryption at rest
• Per-tenant quotas (storage, requests, rate limits)
• TLS (HTTP & gRPC)

Observability

• Admin dashboard
• Prometheus metrics (counters, histograms)
• Request and endpoint statistics
• Structured logging and tracing spans
• Kubernetes health probes

Production-grade Design

• Memory-safe Rust
• Test suite, automated CI
• Documentation and sample config
• Single static binary

🗺️ Roadmap

v0.7.0 (latest)

• Secondary indexes
• Multi-key transactions
• Durable S3-backed object store
• Batch import/export

Next (v0.8.0+)

• Cross-datacenter replication
• Change Data Capture (CDC)
• Admin Web UI
• Backup & Restore
• Plugin system

📖 Story

minikv started as a 24-hour challenge by a Rust learner (82 days into the language!). It now serves as both a playground and a reference for distributed systems, demonstrating curiosity, learning-by-doing, and robust engineering.

📚 Documentation

• Example config : config.example.toml
• Cluster, API, usage : see docs/
• Certificate generation : certs/README.md

🛠️ Development

cargo test           # Run all tests
cargo clippy --fix   # Lint and fix
cargo fmt            # Format code

Continuous Integration runs on push & PR via .github/workflows/ci.yml.

🤝 Contributing

Issues and PRs welcome! See CONTRIBUTING.md.

📬 Contact

• GitHub: whispem/minikv
• Email: via GitHub profile