English | 中文

---

jdb_buf : Thread-Local Buffer Pool with Zero-Copy Cross-Thread Recycling

Table of Contents

• Overview
• Features
• Installation
• Usage
• API Reference
• Architecture
• Design
• Tech Stack
• Directory Structure
• History

Overview

jdb_buf is a high-performance buffer pool designed for database storage engines. It provides thread-local memory allocation with automatic cross-thread recycling, eliminating syscall overhead for Direct I/O operations.

The pool bridges the gap between low-level aligned memory allocation (jdb_alloc) and upper-layer storage logic, serving as the "circulatory system" of the JDB database kernel.

Features

• Three-tier allocation strategy: Local stack → Recycle channel → OS allocation
• Zero-copy recycling: Buffers dropped in any thread flow back to the owner pool
• Compio integration: Page implements IoBuf, IoBufMut, SetBufInit traits
• Graceful degradation: Safe memory release even after pool destruction
• Lock-free design: Uses crossfire high-performance MPSC channel

Installation

cargo add jdb_buf

Usage

Basic Allocation and Recycling

use jdb_buf::LocalPool;

fn main() -> Result<(), jdb_buf::Error> {
  // Create pool with 64-slot recycle channel
  let mut pool = LocalPool::with_cap(64);

  // Allocate 4KB aligned page
  let page = pool.alloc()?;
  assert_eq!(page.cap(), 4096);

  // Page auto-recycles on drop
  drop(page);

  // Move recycled buffers to local stack
  pool.maintain();
  assert_eq!(pool.cached(), 1);

  Ok(())
}

Cross-Thread Recycling

use jdb_buf::LocalPool;
use std::thread;

fn main() -> Result<(), jdb_buf::Error> {
  let mut pool = LocalPool::with_cap(64);
  let page = pool.alloc()?;

  // Move page to another thread
  let handle = thread::spawn(move || {
    // Page dropped here, sent back via channel
    drop(page);
  });

  handle.join().unwrap();

  // Receive recycled buffer
  pool.maintain();
  assert_eq!(pool.cached(), 1);

  Ok(())
}

Custom Page Size

use jdb_buf::LocalPool;

fn main() -> Result<(), jdb_buf::Error> {
  // 8KB pages with 32-slot channel
  let mut pool = LocalPool::new(8192, 32);
  let page = pool.alloc()?;
  assert_eq!(page.cap(), 8192);

  Ok(())
}

Compio I/O Integration

use jdb_buf::LocalPool;
use compio_buf::{IoBuf, IoBufMut};

fn main() -> Result<(), jdb_buf::Error> {
  let mut pool = LocalPool::with_cap(8);
  let mut page = pool.alloc()?;

  // Use as compio buffer
  let ptr = page.as_buf_mut_ptr();
  unsafe { *ptr = 0x42 };

  assert_eq!(page.buf_capacity(), 4096);

  Ok(())
}

API Reference

LocalPool

Thread-local buffer pool.

Method	Description
new(page_size, cap)	Create pool with custom page size and channel capacity
with_cap(cap)	Create pool with default 4KB page size
alloc()	Allocate page (returns Result<Page>)
maintain()	Drain recycle channel to local stack
cached()	Get cached buffer count

Page

Smart buffer with auto-recycle on drop. Implements Deref<Target=RawIoBuf>.

Trait	Description
IoBuf	Read buffer for compio async I/O
IoBufMut	Write buffer for compio async I/O
SetBufInit	Set initialized length after I/O completion
Deref / DerefMut	Access underlying RawIoBuf methods

Exported Data Structures

Error

Error type for buffer operations.

pub enum Error {
  Alloc(jdb_alloc::Error),
}

Result<T>

Alias for std::result::Result<T, Error>.

Architecture

graph TD
    A[LocalPool::alloc] --> B{Local Stack?}
    B -->|Yes| C[Pop from Stack]
    B -->|No| D{Recycle Channel?}
    D -->|Yes| E[try_recv]
    D -->|No| F[AlignedBuf::page]
    C --> G[Page]
    E --> G
    F --> H[into_raw_parts]
    H --> I[RawIoBuf::from_raw_parts]
    I --> G
    G --> J[User Code]
    J --> K[Page::drop]
    K --> L{try_send OK?}
    L -->|Yes| M[Recycle Channel]
    L -->|No| N[free_raw]
    M --> O[LocalPool::maintain]
    O --> P[Local Stack]

Design

Allocation Flow:

1. Check local stack (LIFO, cache-friendly)
2. Try recycle channel (cross-thread returns)
3. Allocate from OS via jdb_alloc

Recycling Flow:

1. Page::drop attempts try_send to channel
2. If channel full/closed, memory freed via AlignedBuf::from_raw_parts
3. maintain() drains channel to local stack

Module Interaction:

graph LR
    A[jdb_alloc] -->|AlignedBuf| B[jdb_buf]
    B -->|RawIoBuf| C[compio-buf]
    B -->|MPSC| D[crossfire]
    
    subgraph jdb_buf
        B --> E[LocalPool]
        E --> F[Page]
        F --> G[IoBuf/IoBufMut]
    end

The Page struct acts as a smart pointer wrapper around RawIoBuf from jdb_alloc, providing automatic lifecycle management and seamless integration with async I/O frameworks through trait implementations.

Tech Stack

Component	Purpose
jdb_alloc	4KB-aligned memory allocation for Direct I/O
crossfire	Lock-free MPSC channel with blocking sender
compio-buf	Async I/O buffer traits

Directory Structure

jdb_buf/
├── src/
│   ├── lib.rs      # LocalPool, Page implementation
│   └── error.rs    # Error types
├── tests/
│   └── main.rs     # Integration tests
└── Cargo.toml

History

The concept of buffer pools dates back to the 1970s when IBM's System R introduced the idea of managing database pages in memory. The term "buffer pool" became standard in database literature after the influential 1981 paper "The Design of POSTGRES" by Michael Stonebraker.

Modern buffer pools face a unique challenge: io_uring and Direct I/O require page-aligned memory, but frequent mmap/munmap syscalls are expensive. Thread-local pools with cross-thread recycling solve this by amortizing allocation costs across many I/O operations.

The "shared-nothing" architecture used here, where each thread owns its pool but accepts returns from other threads, mirrors the design of ScyllaDB and Redpanda—databases that achieve millions of IOPS by eliminating lock contention.

This approach draws inspiration from the buffer management techniques pioneered in the 1990s by the PostgreSQL project, which introduced the concept of buffer pinning and local buffer caches to reduce contention in multi-core systems. The zero-copy recycling mechanism is reminiscent of the slab allocators used in the Linux kernel for efficient memory management.

---

About

This project is an open-source component of js0.site ⋅ Refactoring the Internet Plan.

We are redefining the development paradigm of the Internet in a componentized way. Welcome to follow us:

• Google Group
• js0site.bsky.social

---

jdb_buf : 线程本地缓冲池，支持零拷贝跨线程回收

目录

• 概述
• 特性
• 安装
• 使用
• API 参考
• 架构
• 设计
• 技术栈
• 目录结构
• 历史

概述

jdb_buf 是为数据库存储引擎设计的高性能缓冲池。提供线程本地内存分配与自动跨线程回收，消除 Direct I/O 操作的 syscall 开销。

该模块连接底层对齐内存分配器 (jdb_alloc) 与上层存储逻辑，是 JDB 数据库内核的"血液循环系统"。

特性

• 三级分配策略：本地栈 → 回收通道 → OS 分配
• 零拷贝回收：任意线程释放的缓冲区自动流回所属池
• Compio 集成：Page 实现 IoBuf、IoBufMut、SetBufInit trait
• 优雅降级：池销毁后仍能安全释放内存
• 无锁设计：使用 crossfire 高性能 MPSC 通道

安装

cargo add jdb_buf

使用

基本分配与回收

use jdb_buf::LocalPool;

fn main() -> Result<(), jdb_buf::Error> {
  // 创建池，回收通道容量 64
  let mut pool = LocalPool::with_cap(64);

  // 分配 4KB 对齐页面
  let page = pool.alloc()?;
  assert_eq!(page.cap(), 4096);

  // Page drop 时自动回收
  drop(page);

  // 将回收的缓冲区搬运到本地栈
  pool.maintain();
  assert_eq!(pool.cached(), 1);

  Ok(())
}

跨线程回收

use jdb_buf::LocalPool;
use std::thread;

fn main() -> Result<(), jdb_buf::Error> {
  let mut pool = LocalPool::with_cap(64);
  let page = pool.alloc()?;

  // 将 page 移动到另一线程
  let handle = thread::spawn(move || {
    // Page 在此处 drop，通过通道发回
    drop(page);
  });

  handle.join().unwrap();

  // 接收回收的缓冲区
  pool.maintain();
  assert_eq!(pool.cached(), 1);

  Ok(())
}

自定义页大小

use jdb_buf::LocalPool;

fn main() -> Result<(), jdb_buf::Error> {
  // 8KB 页面，通道容量 32
  let mut pool = LocalPool::new(8192, 32);
  let page = pool.alloc()?;
  assert_eq!(page.cap(), 8192);

  Ok(())
}

Compio I/O 集成

use jdb_buf::LocalPool;
use compio_buf::{IoBuf, IoBufMut};

fn main() -> Result<(), jdb_buf::Error> {
  let mut pool = LocalPool::with_cap(8);
  let mut page = pool.alloc()?;

  // 作为 compio 缓冲区使用
  let ptr = page.as_buf_mut_ptr();
  unsafe { *ptr = 0x42 };

  assert_eq!(page.buf_capacity(), 4096);

  Ok(())
}

API 参考

LocalPool

线程本地缓冲池。

方法	说明
new(page_size, cap)	创建池，指定页大小和通道容量
with_cap(cap)	创建池，默认 4KB 页大小
alloc()	分配页面（返回 Result<Page>）
maintain()	将回收通道数据搬运到本地栈
cached()	获取缓存的缓冲区数量

Page

智能缓冲区，drop 时自动回收。实现 Deref<Target=RawIoBuf>。

Trait	说明
IoBuf	compio 异步 I/O 读缓冲区
IoBufMut	compio 异步 I/O 写缓冲区
SetBufInit	I/O 完成后设置已初始化长度
Deref / DerefMut	访问底层 RawIoBuf 方法

导出的数据结构

Error

缓冲区操作错误类型。

pub enum Error {
  Alloc(jdb_alloc::Error),
}

Result<T>

std::result::Result<T, Error> 的类型别名。

架构

graph TD
    A[LocalPool::alloc] --> B{本地栈?}
    B -->|是| C[从栈弹出]
    B -->|否| D{回收通道?}
    D -->|是| E[try_recv]
    D -->|否| F[AlignedBuf::page]
    C --> G[Page]
    E --> G
    F --> H[into_raw_parts]
    H --> I[RawIoBuf::from_raw_parts]
    I --> G
    G --> J[用户代码]
    J --> K[Page::drop]
    K --> L{try_send 成功?}
    L -->|是| M[回收通道]
    L -->|否| N[free_raw]
    M --> O[LocalPool::maintain]
    O --> P[本地栈]

设计

分配流程：

1. 检查本地栈（LIFO，缓存友好）
2. 尝试回收通道（跨线程返回）
3. 通过 jdb_alloc 从 OS 分配

回收流程：

1. Page::drop 尝试 try_send 到通道
2. 通道满/关闭时，通过 AlignedBuf::from_raw_parts 释放内存
3. maintain() 将通道数据搬运到本地栈

模块交互：

graph LR
    A[jdb_alloc] -->|AlignedBuf| B[jdb_buf]
    B -->|RawIoBuf| C[compio-buf]
    B -->|MPSC| D[crossfire]
    
    subgraph jdb_buf
        B --> E[LocalPool]
        E --> F[Page]
        F --> G[IoBuf/IoBufMut]
    end

Page 结构体作为 RawIoBuf（来自 jdb_alloc）的智能指针包装器，提供自动生命周期管理，并通过 trait 实现与异步 I/O 框架的无缝集成。

技术栈

组件	用途
jdb_alloc	4KB 对齐内存分配，用于 Direct I/O
crossfire	无锁 MPSC 通道，阻塞发送端
compio-buf	异步 I/O 缓冲区 trait

目录结构

jdb_buf/
├── src/
│   ├── lib.rs      # LocalPool、Page 实现
│   └── error.rs    # 错误类型
├── tests/
│   └── main.rs     # 集成测试
└── Cargo.toml

历史

缓冲池的概念可追溯到 1970 年代，IBM System R 首次引入在内存中管理数据库页面的思想。1981 年 Michael Stonebraker 的论文《The Design of POSTGRES》使"缓冲池"成为数据库领域的标准术语。

现代缓冲池面临独特挑战：io_uring 和 Direct I/O 要求页对齐内存，但频繁的 mmap/munmap syscall 开销巨大。线程本地池配合跨线程回收，通过在多次 I/O 操作间分摊分配成本来解决此问题。

此处采用的"Shared-Nothing"架构——每个线程拥有自己的池但接受其他线程的返还——与 ScyllaDB 和 Redpanda 的设计如出一辙。这些数据库通过消除锁竞争实现了百万级 IOPS。

这种方法借鉴了 1990 年代 PostgreSQL 项目开创的缓冲区管理技术，该项目引入了缓冲区固定和本地缓冲区缓存的概念，以减少多核系统中的竞争。零拷贝回收机制让人联想到 Linux 内核中用于高效内存管理的 slab 分配器。

---

关于

本项目为 js0.site ⋅ 重构互联网计划 的开源组件。

我们正在以组件化的方式重新定义互联网的开发范式，欢迎关注：

• 谷歌邮件列表
• js0site.bsky.social