| Crates.io | netabase_store |
| lib.rs | netabase_store |
| version | 0.0.8 |
| created_at | 2025-10-26 22:38:33.310233+00 |
| updated_at | 2025-11-29 13:18:46.880208+00 |
| description | A type-safe, multi-backend key-value storage library for Rust with support for native (Sled, Redb) and WASM (IndexedDB) environments. |
| homepage | |
| repository | https://github.com/newsnet-africa/netabase_store.git |
| max_upload_size | |
| id | 1901966 |
| size | 1,335,737 |
A type-safe, multi-backend key-value storage library for Rust with support for native (Sled, Redb) and WASM (IndexedDB) environments, inspired by native_db.
β οΈ Early Development: This crate is still in early development and will change frequently as it stabilizes. It is not advised to use this in a production environment until it stabilizes.
Add to your Cargo.toml:
[package]
name = "my_project"
version = "0.1.0"
edition = "2024"
# Features must be enabled in your crate for macro-generated code
[features]
default = ["native"]
native = ["netabase_store/native"]
[dependencies]
netabase_store = { version = "0.0.7", features = ["native"] }
# Required dependencies
bincode = { version = "2.0", features = ["serde"] }
serde = { version = "1.0", features = ["derive"] }
strum = { version = "0.27.2", features = ["derive"] }
derive_more = { version = "2.0.1", features = ["from", "try_into", "into"] }
libp2p = "0.56" # Optional, if you would like to use this as a persistent backend for [`libp2p-kad` RecordStore implementation](https://docs.rs/libp2p/latest/libp2p/kad/index.html)
anyhow = "1.0"
The default feature set includes native which provides both Sled and Redb backends. You can customize features for your specific needs.
native - (Recommended) Enables both Sled and Redb backends for desktop/server
sled - Sled backend only (high-performance embedded database)
redb - Redb backend only (memory-efficient, ACID compliant)
redb-zerocopy - Zero-copy Redb variant (maximum performance, requires redb)
wasm - IndexedDB backend for browser/WASM applications
libp2p - Enable libp2p integration for distributed systemsrecord-store - Enable RecordStore trait (requires libp2p)# For desktop/server applications (recommended):
netabase_store = { version = "0.0.6", features = ["native"] }
# For WASM/browser applications:
[target.'cfg(target_arch = "wasm32")'.dependencies]
netabase_store = { version = "0.0.7", default-features = false, features = ["wasm"] }
# For specific backend only:
netabase_store = { version = "0.0.7", features = ["sled"] }
# For zero-copy redb optimization:
netabase_store = { version = "0.0.7", features = ["redb-zerocopy"] }
# For libp2p integration:
netabase_store = { version = "0.0.7", features = ["native", "libp2p"] }
use netabase_store::netabase_definition_module;
use netabase_store::traits::model::NetabaseModelTrait;
#[netabase_definition_module(BlogDefinition, BlogKeys)]
pub mod blog_schema {
use netabase_store::{NetabaseModel, netabase};
#[derive(NetabaseModel, bincode::Encode, bincode::Decode, Clone, Debug, serde::Serialize, serde::Deserialize)]
#[netabase(BlogDefinition)]
pub struct User {
#[primary_key]
pub id: u64,
pub username: String,
#[secondary_key]
pub email: String,
}
#[derive(NetabaseModel, bincode::Encode, bincode::Decode, Clone, Debug, serde::Serialize, serde::Deserialize)]
#[netabase(BlogDefinition)]
pub struct Post {
#[primary_key]
pub id: u64,
pub title: String,
pub content: String,
#[secondary_key]
pub author_id: u64,
}
}
use blog_schema::*;
The unified NetabaseStore provides a consistent API across all backends:
use netabase_store::NetabaseStore;
fn main() -> anyhow::Result<()> {
// Create a store with any backend - easily switch by changing one line!
// Option 1: Sled backend (high-performance)
let store = NetabaseStore::<BlogDefinition, _>::sled("./my_db")?;
// Option 2: Redb backend (memory-efficient, ACID)
// let store = NetabaseStore::<BlogDefinition, _>::redb("./my_db.redb")?;
// Option 3: Temporary store for testing
// let store = NetabaseStore::<BlogDefinition, _>::temp()?;
// Open a tree for users - works identically across all backends
let user_tree = store.open_tree::<User>();
// Insert a user
let user = User {
id: 1,
username: "alice".to_string(),
email: "alice@example.com".to_string(),
};
user_tree.put(user.clone())?;
// Get by primary key - use the generated type
let retrieved = user_tree.get(UserPrimaryKey(1))?.unwrap();
assert_eq!(retrieved.username, "alice");
// Alternative: use primary_key() method
let retrieved2 = user_tree.get(user.primary_key())?.unwrap();
assert_eq!(retrieved2.username, "alice");
// Query by secondary key
let users_by_email = user_tree.get_by_secondary_key(
UserSecondaryKeys::Email(UserEmailSecondaryKey("alice@example.com".to_string()))
)?;
assert_eq!(users_by_email.len(), 1);
// Iterate over all users
for result in user_tree.iter() {
let (_key, user) = result?;
println!("User: {} - {}", user.username, user.email);
}
// Access backend-specific features when needed
store.flush()?; // Sled-specific method
Ok(())
}
You can also use backends directly for backend-specific features:
use netabase_store::databases::sled_store::SledStore;
use netabase_store::databases::redb_store::RedbStore;
// Direct Sled usage
let sled_store = SledStore::<BlogDefinition>::temp()?;
let user_tree = sled_store.open_tree::<User>();
// Direct Redb usage
let redb_store = RedbStore::<BlogDefinition>::new("my_database.redb")?;
let user_tree = redb_store.open_tree::<User>();
// Both have identical APIs via NetabaseTreeSync trait
use netabase_store::databases::indexeddb_store::IndexedDBStore;
use netabase_store::traits::tree::NetabaseTreeAsync;
#[cfg(target_arch = "wasm32")]
async fn wasm_example() -> Result<(), Box<dyn std::error::Error>> {
// Create a store in the browser
let store = IndexedDBStore::<BlogDefinition>::new("my_database").await?;
// Note: WASM uses async API
let user_tree = store.open_tree::<User>();
let user = User {
id: 1,
username: "charlie".to_string(),
email: "charlie@example.com".to_string(),
};
// All operations are async
user_tree.put(user.clone()).await?;
let retrieved = user_tree.get(user.primary_key()).await?;
Ok(())
}
All backends share the same core API through traits, but have different characteristics:
let store = NetabaseStore::<BlogDefinition, _>::sled("./my_db")?;
let tree = store.open_tree::<User>();
tree.put(user)?; // Synchronous operations
let store = NetabaseStore::<BlogDefinition, _>::redb("./my_db.redb")?;
let tree = store.open_tree::<User>();
tree.put(user)?; // Synchronous operations
Type: Native, persistent, explicit transactions
Use Cases: Maximum performance, explicit transaction control
Strengths:
API Difference: Requires explicit transaction management
use netabase_store::databases::redb_zerocopy::RedbStoreZeroCopy;
let store = RedbStoreZeroCopy::<BlogDefinition>::new("./my_db.redb")?;
// Must use explicit transactions
let mut txn = store.begin_write()?;
let mut tree = txn.open_tree::<User>()?;
tree.put(user)?;
drop(tree);
txn.commit()?; // Must explicitly commit
let store = IndexedDBStore::<BlogDefinition>::new("my_db").await?;
let tree = store.open_tree::<User>();
tree.put(user).await?; // Note: async operations
let result = tree.get(user.primary_key()).await?;
| Backend | Persistence | Async | Best For | Avoid If |
|---|---|---|---|---|
| Sled | β Disk | β Sync | General purpose, high writes | WASM target |
| Redb | β Disk | β Sync | Read-heavy, low memory | Need fastest writes |
| RedbZeroCopy | β Disk | β Sync | Bulk ops, transaction control | Want simple API |
| IndexedDB | β Browser | β Async | Web/WASM apps | Native targets |
Performance Notes:
store.read() and store.write()API Compatibility:
.await) for all operationsThe new unified configuration system provides consistent backend initialization across all database types:
use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;
use netabase_store::databases::sled_store::SledStore;
// Method 1: Builder pattern (recommended)
let config = FileConfig::builder()
.path("app_data.db".into())
.cache_size_mb(1024)
.truncate(true)
.build();
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;
// Method 2: Simple constructor
let config = FileConfig::new("app_data.db");
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::open(config)?;
// Method 3: Temporary database
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::temp()?;
The power of the configuration API is that you can switch backends without changing your code:
use netabase_store::config::FileConfig;
use netabase_store::traits::backend_store::BackendStore;
let config = FileConfig::builder()
.path("my_app.db".into())
.cache_size_mb(512)
.build();
// Try different backends - same config!
#[cfg(feature = "sled")]
let store = <SledStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config.clone())?;
#[cfg(feature = "redb")]
let store = <RedbStore<BlogDefinition> as BackendStore<BlogDefinition>>::new(config.clone())?;
#[cfg(feature = "redb-zerocopy")]
let store = <RedbStoreZeroCopy<BlogDefinition> as BackendStore<BlogDefinition>>::new(config)?;
// All have the same API from this point on!
let user_tree = store.open_tree::<User>();
FileConfig (for Sled, Redb, RedbZeroCopy):
path: PathBuf - Database file/directory path
cache_size_mb: usize - Cache size in megabytes (default: 256)
create_if_missing: bool - Create if doesn't exist (default: true)
truncate: bool - Delete existing data (default: false)
read_only: bool - Open read-only (default: false)
use_fsync: bool - Fsync for durability (default: true)
IndexedDBConfig (for WASM):
database_name: String - IndexedDB database nameversion: u32 - Schema version (default: 1)For high-performance bulk operations, use the transaction API (10-100x faster than individual operations):
use netabase_store::NetabaseStore;
let store = NetabaseStore::<BlogDefinition, _>::sled("./my_db")?;
// Create a write transaction for bulk operations
// NOTE: write() returns TxnGuard directly, not a Result
let mut txn = store.write();
let mut user_tree = txn.open_tree::<User>();
// Bulk insert - 8-9x faster than individual puts!
let users: Vec<User> = (0..1000)
.map(|i| User {
id: i,
username: format!("user{}", i),
email: format!("user{}@example.com", i),
})
.collect();
// All inserts in a single transaction
user_tree.put_many(users)?;
// Bulk read within transaction
let keys: Vec<UserPrimaryKey> = (0..100).map(UserPrimaryKey).collect();
let users: Vec<Option<User>> = user_tree.get_many(keys)?;
// Commit all changes atomically
txn.commit()?;
Transaction Methods:
put_many(Vec<M>) - Insert multiple models in one transactionget_many(Vec<M::PrimaryKey>) - Read multiple models in one transactionBackend Support:
For more examples, see examples/batch_operations_all_backends.rs
Or use the batch API for more control:
use netabase_store::traits::batch::Batchable;
// Create a batch
let mut batch = user_tree.create_batch()?;
// Add many operations
for i in 0..1000 {
batch.put(User { /* ... */ })?;
}
// Commit atomically - all or nothing
batch.commit()?;
The batch API provides fine-grained control and is supported on both sync backends (Sled, Redb).
For maximum performance and atomicity, use the transaction API to reuse a single transaction across multiple operations:
use netabase_store::NetabaseStore;
let store = NetabaseStore::<BlogDefinition, _>::sled("./my_db")?;
// Read-only transaction - multiple concurrent reads allowed
// NOTE: read() and write() return guards directly, not Results
let txn = store.read();
let user_tree = txn.open_tree::<User>();
let user = user_tree.get(UserPrimaryKey(1))?;
// Transaction auto-closes on drop
// Read-write transaction - exclusive access, atomic commit
let mut txn = store.write();
let mut user_tree = txn.open_tree::<User>();
// All operations share the same transaction
for i in 0..1000 {
let user = User {
id: i,
username: format!("user{}", i),
email: format!("user{}@example.com", i),
};
user_tree.put(user)?;
}
// Bulk helpers also work within transactions
user_tree.put_many(more_users)?;
// Commit all changes atomically
txn.commit()?;
// Or drop without committing to rollback
Transaction Benefits:
Compile-Time Safety:
let txn = store.read(); // ReadOnly transaction
let tree = txn.open_tree::<User>();
tree.put(user)?; // β Compile error: put() not available on ReadOnly!
Backend Support Notes:
store.read() and store.write() transaction APIstore.begin_write() and txn.commit() pattern (different API)examples/transactions.rs for detailed examplesSecondary keys enable efficient lookups on non-primary fields:
#[derive(NetabaseModel, Clone, bincode::Encode, bincode::Decode)]
#[netabase(BlogDefinition)]
pub struct Article {
#[primary_key]
pub id: u64,
pub title: String,
#[secondary_key]
pub category: String,
#[secondary_key]
pub published: bool,
}
// Query by single secondary key
let tech_articles = article_tree
.get_by_secondary_key(
ArticleSecondaryKeys::Category(
ArticleCategorySecondaryKey("tech".to_string())
)
)?;
// Bulk query multiple secondary keys (2-3x faster!)
let keys = vec![
ArticleSecondaryKeys::Category(ArticleCategorySecondaryKey("tech".to_string())),
ArticleSecondaryKeys::Category(ArticleCategorySecondaryKey("science".to_string())),
];
let results: Vec<Vec<Article>> = article_tree.get_many_by_secondary_keys(keys)?;
// results[0] = tech articles, results[1] = science articles
use netabase_store::NetabaseStore;
let store = NetabaseStore::<BlogDefinition, _>::sled("blog_db")?;
// Different trees for different models
let user_tree = store.open_tree::<User>();
let post_tree = store.open_tree::<Post>();
// Each tree is independent but shares the same underlying database
user_tree.put(user)?;
post_tree.put(post)?;
use netabase_store::NetabaseStore;
// Perfect for unit tests - no I/O, no cleanup needed
let store = NetabaseStore::<BlogDefinition, _>::temp()?;
let user_tree = store.open_tree::<User>();
user_tree.put(user)?;
It is technically possible to implement a backend of your own, but much of this implementation needed to be generated with macros, with a few implementation specific quirks. This makes it a bit hard for me to track exactly how to create reproducable instructions, but you can do one of 2 things:
netabase_macros to see what gents generated and how things string upkivis or native_model are probably better suited.Netabase Store is designed for high performance while maintaining type safety. The library provides multiple APIs optimized for different use cases, with comprehensive benchmarking and profiling support.
The library offers three APIs with different performance characteristics:
put_many(), get_many(), get_many_by_secondary_keys() - single transaction for multiple itemsComprehensive benchmarks comparing all implementations across multiple dataset sizes (10, 100, 500, 1000, 5000 items):
| Implementation | Time | vs Raw | Notes |
|---|---|---|---|
| Raw Redb (baseline) | 1.42 ms | 0% | Single transaction, manual index management |
| Wrapper Redb (bulk) | 3.10 ms | +118% | put_many() - single transaction |
| Wrapper Redb (loop) | 27.3 ms | +1,822% | Individual put() calls - creates N transactions |
| ZeroCopy (bulk) | 3.51 ms | +147% | put_many() with explicit transaction |
| ZeroCopy (loop) | 4.34 ms | +206% | Loop with single explicit transaction |
Key Insights:
| Implementation | Time | vs Raw | Notes |
|---|---|---|---|
| Raw Redb (baseline) | 164 Β΅s | 0% | Single transaction |
| Wrapper Redb (bulk) | 382 Β΅s | +133% | get_many() - single transaction |
| Wrapper Redb (loop) | 895 Β΅s | +446% | Individual get() calls - creates N transactions |
| ZeroCopy (single txn) | 692 Β΅s | +322% | Explicit read transaction |
Key Insights:
get_many() provides 2.3x speedup over individual gets (895Β΅s β 382Β΅s)| Implementation | Time | vs Raw | Notes |
|---|---|---|---|
| Raw Redb (baseline) | 291 Β΅s | 0% | 10 transactions, manual index traversal |
| Wrapper Redb (bulk) | 470 Β΅s | +61% | get_many_by_secondary_keys() - single transaction |
| Wrapper Redb (loop) | 1.02 ms | +248% | 10 separate get_by_secondary_key() calls |
| ZeroCopy (single txn) | 5.41 Β΅s | -98% | Single transaction, optimized index access |
Key Insights:
// β Slow: Creates 1000 transactions
for user in users {
tree.put(user)?; // Each call = new transaction
}
// β
Fast: Single transaction
tree.put_many(users)?; // 8-9x faster!
Available Bulk Methods:
put_many(Vec<M>) - Bulk insertget_many(Vec<M::Keys>) - Bulk readget_many_by_secondary_keys(Vec<SecondaryKey>) - Bulk secondary queries// For write-heavy workloads
// NOTE: write() returns TxnGuard directly, not a Result
let mut txn = store.write();
let mut tree = txn.open_tree::<User>();
for user in users {
tree.put(user)?; // All share same transaction
}
txn.commit()?; // Single atomic commit
| Use Case | Recommended API | Reason |
|---|---|---|
| Simple CRUD, few operations | Standard wrapper | Simplest API, auto-commit |
| Bulk inserts/reads (100+ items) | Bulk methods | 8-9x faster than loops |
| Complex transactions | Explicit transactions | Full control, atomic commits |
| Read-heavy queries | ZeroCopy API | Up to 54x faster for secondary queries |
# Cross-store comparison (all backends, multiple sizes)
cargo bench --bench cross_store_comparison --features native
# Generate visualizations
uv run scripts/generate_benchmark_charts.py
# View results
open docs/benchmarks/insert_comparison_bars.png
open docs/benchmarks/overhead_percentages.png
open docs/benchmarks/bulk_api_speedup.png
The wrapper APIs prioritize type safety and ergonomics. For applications where the overhead is significant:
The read-path overhead in Redb comes from type system limitations with Generic Associated Types (GATs). We prioritize safety over unsafe transmutes. For applications where this matters:
See benchmark results and visualizations in docs/benchmarks/ for detailed performance analysis.
# Run all tests
cargo test --all-features
# Run native tests only
cargo test --features native
# Run WASM tests (requires wasm-pack and Firefox)
wasm-pack test --headless --firefox --features wasm
See ARCHITECTURE.md for a deep dive into the library's design.
βββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Your Application Code β
β (Type-safe models with macros) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
β
βββββββββββββββββββββββββββββββββββββββββββββββββββββ
β NetabaseStore<D, Backend> β
β (Unified API layer - Recommended) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββΌββββββββββββββ
β β β
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β SledStore β β RedbStore β βIndexedDBStoreβ
β <D> β β <D> β β <D> β
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β β β
β β β
βββββββββββββββββββββββββββββββββββββββββββββββββββ
β Trait Layer β
β (NetabaseTreeSync, NetabaseTreeAsync) β
β (OpenTree, Batchable, StoreOps) β
βββββββββββββββββββββββββββββββββββββββββββββββββββ
β β β
β β β
βββββββ ββββββββ βββββββββββ
β Sledβ β Redb β βIndexedDBβ
βββββββ ββββββββ βββββββββββ
Native Native WASM
redb-zerocopy feature) - Phase 1 CompleteSee the test_netabase_store_usage crate for a complete working example.
examples/basic_store.rs - Basic CRUD operations with Sled
examples/unified_api.rs - Working with the NetabaseStore unified API
examples/config_api_showcase.rs - Configuration system and backend switching
examples/batch_operations.rs - Batch operations (Sled-focused)examples/transactions.rs - Transaction API (Sled-focused)examples/redb_basic.rs - Redb-specific featuresexamples/redb_zerocopy.rs - RedbZeroCopy explicit transaction APIexamples/subscription_demo.rs - Change notification systemexamples/subscription_streams.rs - Advanced streaming examplestests/backend_crud_tests.rs - Comprehensive CRUD tests for all backendstests/wasm_tests.rs - WASM/IndexedDB usage patternstests/comprehensive_store_tests.rs - Full test suite# Run multi-backend examples (automatically includes available backends)
cargo run --example batch_operations_all_backends --features native
cargo run --example transactions_all_backends --features native
# Run with specific backends
cargo run --example batch_operations_all_backends --features "sled,redb"
cargo run --example transactions_all_backends --features "sled"
# Run single-backend examples
cargo run --example basic_store --features native
cargo run --example redb_zerocopy --features redb-zerocopy
Note: Examples work with both Sled and Redb backends. Switch backends by changing the initialization method.
I was mostly trying to abstract a persisten backend for libp2p::kad::store, and got carried away.
Contributions are welcome! Feel free to leave a PR.
This project is licensed under the GPL-3.0-only License - see the LICENSE file for details.
Built with: