flashkv
| Crates.io | flashkv |
| lib.rs | flashkv |
| version | 0.1.0 |
| created_at | 2025-11-28 10:12:39.923081+00 |
| updated_at | 2025-11-28 10:12:39.923081+00 |
| description | A high-performance, in-memory Key-Value database built in Rust |
| homepage | https://github.com/Ariz/flashkv |
| repository | https://github.com/Ariz/flashkv |
| max_upload_size | |
| id | 1955086 |
| size | 235,219 |
Ariz (arino08)
documentation
https://docs.rs/flashkv
README
███████████ ████ █████ █████ ████ █████ █████ ░░███░░░░░░█░░███ ░░███ ░░███ ███░ ░░███ ░░███ ░███ █ ░ ░███ ██████ █████ ░███████ ░███ ███ ░███ ░███ ░███████ ░███ ░░░░░███ ███░░ ░███░░███ ░███████ ░███ ░███ ░███░░░█ ░███ ███████ ░░█████ ░███ ░███ ░███░░███ ░░███ ███ ░███ ░ ░███ ███░░███ ░░░░███ ░███ ░███ ░███ ░░███ ░░░█████░ █████ █████░░████████ ██████ ████ █████ █████ ░░████ ░░███ ░░░░░ ░░░░░ ░░░░░░░░ ░░░░░░ ░░░░ ░░░░░ ░░░░░ ░░░░ ░░░
A High-Performance, Redis-Compatible In-Memory Database Built from Scratch in Rust
A learning-focused systems programming project demonstrating database internals, network protocols, and concurrent programming in Rust
Features • Architecture • Getting Started • Commands • What I Learned
About This Project
FlashKV is a Redis-compatible, in-memory key-value database that I built from scratch to deeply understand systems programming concepts. This isn't just another Redis clone—it's a comprehensive learning journey through database internals, network programming, protocol design, and concurrent systems.
Why I Built This
I wanted to go beyond tutorials and actually implement the core systems that power modern databases:
- How do databases handle thousands of concurrent connections? → Built an async TCP server with Tokio
- How do protocols like Redis communicate efficiently? → Implemented a zero-copy RESP parser
- How do you make data structures thread-safe without killing performance? → Designed a sharded storage engine
- How do databases handle key expiration? → Created a dual-strategy expiry system
💡 This project comes with 15 detailed documentation files explaining every design decision, from Rust fundamentals to advanced concurrency patterns.
Features
Core Database Features
| Feature | Description |
|---|---|
| Redis Protocol Compatible | Works with redis-cli, Telnet, and any Redis client library |
| Thread-Safe Concurrent Access | 64-shard architecture allowing parallel reads/writes |
| TTL & Auto-Expiry | Keys can expire automatically with lazy + active cleanup |
| Multiple Data Types | Strings and Lists with full Redis-compatible operations |
| Pattern Matching | KEYS command with glob-style pattern support (*, ?, [abc]) |
| Built-in Statistics | Real-time metrics for ops/second, memory usage, and more |
Technical Highlights
| Component | Implementation |
|---|---|
| Async Runtime | Tokio for handling 10,000+ concurrent connections |
| Zero-Copy Parsing | bytes::Bytes for efficient memory handling |
| Protocol | Full RESP (Redis Serialization Protocol) + inline commands |
| Concurrency | Arc<RwLock> with sharding to minimize contention |
| Background Tasks | Adaptive expiry sweeper that adjusts based on load |
Architecture
┌────────────────────────────────────────────────────────────────────────────┐
│ FlashKV │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ TCP Server │────▶│ Connection │────▶│ Command │ │
│ │ (Listener) │ │ Handler │ │ Handler │ │
│ │ │ │ │ │ │ │
│ │ • Accepts │ │ • Per-client │ │ • Parses │ │
│ │ connections│ │ read loop │ │ commands │ │
│ │ • Spawns │ │ • Buffered │ │ • Dispatches │ │
│ │ tasks │ │ I/O │ │ to storage │ │
│ └──────────────┘ └──────────────┘ └──────┬───────┘ │
│ │ │
│ ┌─────────────────────────────────────────┼────────────────┐ │
│ │ ▼ │ │
│ │ ┌────────────────────────────────────────────────────┐ │ │
│ │ │ StorageEngine │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │
│ │ │ │ Shard 0 │ │ Shard 1 │ │ Shard 2 │ │ ... │ │ │ │
│ │ │ │ ─────── │ │ ─────── │ │ ─────── │ │ 64 │ │ │ │
│ │ │ │ RwLock │ │ RwLock │ │ RwLock │ │ shards │ │ │ │
│ │ │ │ HashMap │ │ HashMap │ │ HashMap │ │ │ │ │ │
│ │ │ │(strings)│ │(strings)│ │(strings)│ │ │ │ │ │
│ │ │ │ HashMap │ │ HashMap │ │ HashMap │ │ │ │ │ │
│ │ │ │ (lists) │ │ (lists) │ │ (lists) │ │ │ │ │ │
│ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ │
│ │ │ │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ │ ▲ │ │
│ │ │ │ │
│ │ ┌────────────────────────┴───────────────────────────┐ │ │
│ │ │ ExpirySweeper (Background) │ │ │
│ │ │ • Adaptive interval (100ms - 1s based on load) │ │ │
│ │ │ • Scans shards for expired keys │ │ │
│ │ │ • Graceful shutdown support │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ RESP Protocol Parser │ │
│ │ • Zero-copy parsing with bytes::Bytes │ │
│ │ • Supports: Simple Strings, Errors, Integers, Bulk Strings, Arrays │ │
│ │ • Inline command support (plain text like "SET key value") │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────────┘
Key Design Decisions
| Decision | Why |
|---|---|
| 64 Shards | Reduces lock contention—keys are distributed by hash, allowing parallel access |
| RwLock per Shard | Multiple readers can access data simultaneously; writers get exclusive access |
| Separate List Storage | Type safety—prevents accidental string operations on lists |
| Lazy + Active Expiry | Lazy catches expired keys on access; active reclaims memory for untouched keys |
| VecDeque for Lists | O(1) push/pop on both ends, perfect for LPUSH/RPUSH/LPOP/RPOP |
Getting Started
Prerequisites
- Rust 1.75+ (Install Rust)
Building
# Clone the repository
git clone https://github.com/yourusername/flashkv.git
cd flashkv
# Build in release mode (optimized)
cargo build --release
# Run tests (69 tests covering all functionality)
cargo test
Running the Server
# Start with defaults (127.0.0.1:6379)
cargo run --release
# Or with custom settings
./target/release/flashkv --host 0.0.0.0 --port 6380
Connecting
Option 1: Using redis-cli
redis-cli -p 6379
127.0.0.1:6379> PING
PONG
127.0.0.1:6379> SET greeting "Hello, World!"
OK
127.0.0.1:6379> GET greeting
"Hello, World!"
Option 2: Using Telnet (inline commands)
telnet localhost 6379
set name Ariz
+OK
get name
$4
Ariz
Supported Commands
String Commands (17 commands)
| Command | Syntax | Description |
|---|---|---|
SET |
SET key value [EX s] [PX ms] [NX|XX] [GET] |
Set a key with optional expiry and conditions |
GET |
GET key |
Get value by key |
DEL |
DEL key [key ...] |
Delete one or more keys |
EXISTS |
EXISTS key [key ...] |
Check if keys exist |
INCR |
INCR key |
Increment integer value by 1 |
INCRBY |
INCRBY key delta |
Increment by specified amount |
DECR |
DECR key |
Decrement integer value by 1 |
DECRBY |
DECRBY key delta |
Decrement by specified amount |
APPEND |
APPEND key value |
Append to existing string |
STRLEN |
STRLEN key |
Get string length |
MSET |
MSET k1 v1 [k2 v2 ...] |
Set multiple keys atomically |
MGET |
MGET k1 [k2 ...] |
Get multiple values |
SETNX |
SETNX key value |
Set only if key doesn't exist |
SETEX |
SETEX key seconds value |
Set with expiry in seconds |
PSETEX |
PSETEX key ms value |
Set with expiry in milliseconds |
GETSET |
GETSET key value |
Set new value, return old |
GETDEL |
GETDEL key |
Get value and delete key |
List Commands (9 commands)
| Command | Syntax | Description |
|---|---|---|
LPUSH |
LPUSH key val [val ...] |
Push to head of list |
RPUSH |
RPUSH key val [val ...] |
Push to tail of list |
LPOP |
LPOP key |
Remove and return from head |
RPOP |
RPOP key |
Remove and return from tail |
LLEN |
LLEN key |
Get list length |
LINDEX |
LINDEX key index |
Get element by index (supports negative) |
LRANGE |
LRANGE key start stop |
Get range of elements |
LSET |
LSET key index value |
Set element at index |
LREM |
LREM key count value |
Remove elements by value |
Key Commands (10 commands)
| Command | Syntax | Description |
|---|---|---|
EXPIRE |
EXPIRE key seconds |
Set TTL in seconds |
PEXPIRE |
PEXPIRE key ms |
Set TTL in milliseconds |
EXPIREAT |
EXPIREAT key timestamp |
Set expiry at Unix timestamp |
TTL |
TTL key |
Get remaining TTL in seconds |
PTTL |
PTTL key |
Get remaining TTL in milliseconds |
PERSIST |
PERSIST key |
Remove expiry from key |
KEYS |
KEYS pattern |
Find keys matching pattern |
TYPE |
TYPE key |
Get type (string/list/none) |
RENAME |
RENAME key newkey |
Rename a key |
RENAMENX |
RENAMENX key newkey |
Rename only if new key doesn't exist |
Server Commands (10 commands)
| Command | Syntax | Description |
|---|---|---|
PING |
PING [message] |
Test connection |
ECHO |
ECHO message |
Echo message back |
INFO |
INFO [section] |
Server information |
DBSIZE |
DBSIZE |
Number of keys |
FLUSHDB |
FLUSHDB |
Clear entire database |
FLUSHALL |
FLUSHALL |
Clear entire database |
COMMAND |
COMMAND |
List available commands |
CONFIG |
CONFIG GET param |
Get configuration |
TIME |
TIME |
Server time |
DEBUG |
DEBUG SLEEP seconds |
Debug utilities |
Project Structure
flashkv/
├── src/
│ ├── main.rs # Entry point, CLI parsing, TCP server setup
│ ├── lib.rs # Public API exports
│ │
│ ├── protocol/ # RESP Protocol Implementation
│ │ ├── mod.rs # Module exports
│ │ ├── types.rs # RespValue enum, serialization
│ │ └── parser.rs # Zero-copy parser, inline command support
│ │
│ ├── storage/ # Storage Engine
│ │ ├── mod.rs # Module exports
│ │ ├── engine.rs # Sharded HashMap, Entry/ListEntry, all operations
│ │ └── expiry.rs # Background sweeper task
│ │
│ ├── commands/ # Command Handlers
│ │ ├── mod.rs # Module exports
│ │ └── handler.rs # 46 command implementations
│ │
│ └── connection/ # Connection Management
│ ├── mod.rs # Module exports
│ └── handler.rs # Per-client read loop, stats
│
├── docs/ # Comprehensive Documentation (15 files)
│ ├── 00_INDEX.md # Documentation index
│ ├── 01_RUST_FUNDAMENTALS.md # Rust concepts used in the project
│ ├── 02_ASYNC_PROGRAMMING.md # Tokio, async/await patterns
│ ├── 03_NETWORKING_BASICS.md # TCP, sockets, buffered I/O
│ ├── 04_RESP_PROTOCOL.md # Redis protocol specification
│ ├── 05_PROTOCOL_TYPES.md # RespValue implementation
│ ├── 06_PROTOCOL_PARSER.md # Parser design and zero-copy
│ ├── 07_CONCURRENCY.md # Arc, RwLock, sharding strategies
│ ├── 08_STORAGE_ENGINE.md # Storage design and operations
│ ├── 09_EXPIRY_SWEEPER.md # Background task implementation
│ ├── 10_COMMAND_HANDLER.md # Command dispatch and handlers
│ ├── 11_CONNECTION_HANDLER.md# Client connection lifecycle
│ ├── 12_MAIN_SERVER.md # Server bootstrap and shutdown
│ ├── 13_LIB_EXPORTS.md # Public API design
│ ├── 14_BENCHMARKING.md # Performance testing guide
│ └── 15_EXERCISES.md # Extension exercises
│
├── benches/
│ └── throughput.rs # Criterion benchmarks
│
├── Cargo.toml # Dependencies and metadata
└── README.md # You are here!
What I Learned
Building FlashKV taught me deep, practical knowledge in several areas:
Systems Programming
| Concept | What I Implemented |
|---|---|
| Memory Management | Zero-copy parsing with bytes::Bytes, avoiding allocations in hot paths |
| Concurrency | Sharded locks allowing parallel access without data races |
| Background Tasks | Graceful spawning and shutdown of async tasks |
| Resource Cleanup | Proper cleanup on connection drop and server shutdown |
Network Programming
| Concept | What I Implemented |
|---|---|
| TCP Servers | Async listener accepting thousands of connections |
| Buffered I/O | Efficient reading with BytesMut buffers |
| Protocol Parsing | Incremental parser handling partial data |
| Connection Lifecycle | Per-client state management |
Database Internals
| Concept | What I Implemented |
|---|---|
| Storage Engines | Thread-safe HashMap with TTL support |
| Data Types | Strings and Lists with type-safe operations |
| Expiry Mechanisms | Lazy deletion + active background sweeping |
| Statistics Tracking | Atomic counters for real-time metrics |
Rust Proficiency
| Concept | Where I Used It |
|---|---|
| Ownership & Borrowing | Throughout—especially in the parser and storage |
| Lifetimes | Parser returning references into buffers |
| Traits | Debug, Clone, Default implementations |
| Error Handling | Result, Option, thiserror for custom errors |
| Async/Await | Tokio-based server and background tasks |
| Smart Pointers | Arc, Box for shared ownership |
| Interior Mutability | RwLock for concurrent access |
Performance
Run benchmarks:
cargo bench
Or use Redis's benchmark tool:
# Start FlashKV
cargo run --release &
# Run benchmark
redis-benchmark -t set,get,incr -n 100000 -q -p 6379
Expected results on modern hardware:
SET: ~150,000 requests/second
GET: ~200,000 requests/second
INCR: ~180,000 requests/second
Future Enhancements
The following features could be added to extend FlashKV:
- Persistence - RDB snapshots and AOF logging
- Pub/Sub - Publish/Subscribe messaging
- Transactions - MULTI/EXEC command blocks
- More Data Types - Sets, Sorted Sets, Hashes
- Cluster Mode - Distributed sharding across nodes
- Lua Scripting - Server-side scripting support
Documentation
This project includes 15 detailed documentation files covering:
- Rust Fundamentals - Ownership, borrowing, lifetimes
- Async Programming - Tokio, futures, task spawning
- Networking - TCP, buffered I/O, connection handling
- Protocol Design - RESP specification and parsing
- Concurrency - Locks, atomics, sharding strategies
- Storage Design - Data structures, expiry, statistics
- And more...
Each file includes code examples, diagrams, and exercises.
Contributing
Contributions are welcome! Whether it's:
- Bug fixes
- New features
- Documentation improvements
- Additional tests
License
MIT License - feel free to use this for learning and building!
Built with ❤️ and 🦀 Rust
This project was built as a learning exercise to understand database internals, network programming, and systems design.