MemBase
_{^{memory data mapping}}

Ultra-high-performance memmap libraryå for Rust

     

Performance Benchmarks

Sequential Read Performance:

• Standard I/O: 2,784 MB/s
• MemBase: 26,277 MB/s
• 9.4x faster than traditional I/O

Random Access Performance:

• Standard I/O: 397,634 ops/s
• MemBase: 1,509,185 ops/s
• 3.8x faster random access

Key Features

Zero-Copy Architecture

Direct memory access eliminates buffer copying and system call overhead, delivering raw performance for data-intensive applications.

Advanced Memory Management

Intelligent memory mapping strategies optimize for both sequential and random access patterns, automatically adapting to workload characteristics.

Cross-Platform Compatibility

Native support across Windows, macOS, and Linux with platform-specific optimizations under a unified API.

Safety-First Design

Memory-safe operations with Rust's ownership model, preventing common pitfalls like buffer overflows and use-after-free errors.

Integrated Database Operations

Built-in database functionality with optimized query execution and in-place file modifications for persistent data structures.

Use Cases

• High-Frequency Trading Systems - Microsecond-critical market data processing
• Database Engines - Storage layer optimization for OLTP and OLAP workloads
• Real-Time Analytics - Stream processing with persistent state management
• Game Engines - Asset loading and world state persistence
• Scientific Computing - Large dataset manipulation and analysis
• Log Processing - High-throughput log ingestion and analysis

Architecture

MemBase leverages OS-level memory mapping primitives with intelligent prefetching and cache-aware algorithms. The library automatically handles:

• Page fault optimization
• Memory pressure management
• Multi-threaded access coordination
• Atomic operations for concurrent modifications

Usage

Add this to your Cargo.toml:

[dependencies]
MemMap = "0.2.0"

Basic Example

use MemMap::{MmapOptions, Mmap};
use std::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open a file
    let file = File::open("data.bin")?;

    // Create a memory map with default options
    let map = unsafe { MmapOptions::new().map(&file)? };

    // Access the memory map
    if map.len() >= 8 {
        let value = unsafe { *(map.as_ptr() as *const u64) };
        println!("First 8 bytes as u64: {}", value);
    }

    Ok(())
}

Advanced Example

use MemMap::{MmapOptions, Mmap, HugePageSize, PrefetchStrategy};
use MemMap::platform::Advice;
use std::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open a file
    let file = File::open("large_data.bin")?;

    // Create a memory map with advanced options
    let map = unsafe {
        MmapOptions::new()
            .read(true)
            .write(true)
            // Use huge pages if available
            .huge_pages(HugePageSize::TwoMB)
            // Use sequential prefetching
            .prefetch(PrefetchStrategy::Sequential)
            // Populate the mapping immediately
            .populate(true)
            // Map the file
            .map_mut(&file)?
    };

    // Advise the kernel about our access pattern
    map.advise(Advice::Sequential)?;

    // Use the memory map
    // ...

    // Flush changes to disk
    map.flush()?;

    Ok(())
}