smartbuf

A high-performance buffered reader with background thread pre-fetching and full seek support.

SmartBuf wraps any Read + Seek implementation and provides:

• Off-thread pre-fetch buffering for improved read performance
• Full seek support with optimization for seeks within buffered data
• Configurable buffer sizes and queue lengths for fine-tuning performance

Features

• Background thread pre-fetching: Data is read ahead of time in a background thread, reducing blocking on I/O operations
• Intelligent seek optimization: Seeks within the current buffer are handled instantly without touching the underlying reader
• Configurable performance: Adjust buffer size and queue length based on your use case
• Standard trait implementation: Implements std::io::Read and std::io::Seek for drop-in compatibility

Installation

Add this to your Cargo.toml:

[dependencies]
smartbuf = "0.1.0"

Quick Example

use smartbuf::SmartBuf;
use std::io::{Read, Seek, SeekFrom, Cursor};

let data = b"Hello, world! This is a test.";
let cursor = Cursor::new(data);
let mut reader = SmartBuf::new(cursor);

// Read some data
let mut buf = vec![0; 5];
reader.read(&mut buf).unwrap();
assert_eq!(&buf, b"Hello");

// Seek back to the beginning
reader.seek(SeekFrom::Start(0)).unwrap();

// Read again
let mut buf = vec![0; 5];
reader.read(&mut buf).unwrap();
assert_eq!(&buf, b"Hello");

// Seek forward
reader.seek(SeekFrom::Current(7)).unwrap();
let mut buf = vec![0; 4];
reader.read(&mut buf).unwrap();
assert_eq!(&buf, b"orld");

Usage

Basic Usage

use smartbuf::SmartBuf;
use std::io::{Read, Seek, SeekFrom};
use std::fs::File;

let file = File::open("data.bin")?;
let mut reader = SmartBuf::new(file);

// Read data
let mut buffer = vec![0; 1024];
let bytes_read = reader.read(&mut buffer)?;

// Seek to a specific position
reader.seek(SeekFrom::Start(1000))?;

// Continue reading from the new position
reader.read(&mut buffer)?;

Custom Buffer Configuration

For fine-tuned performance, you can specify the buffer size and queue length:

use smartbuf::SmartBuf;
use std::io::Cursor;

let data = vec![0u8; 1024 * 1024]; // 1MB of data
let cursor = Cursor::new(data);

// Create with custom buffer size (16KB) and queue length (4)
let mut reader = SmartBuf::with_capacity(16 * 1024, 4, cursor);

// Larger buffers and more queue slots can improve throughput
// at the cost of increased memory usage

Seeking Operations

SmartBuf supports all standard seek operations:

use smartbuf::SmartBuf;
use std::io::{Read, Seek, SeekFrom, Cursor};

let data: Vec<u8> = (0..100).collect();
let cursor = Cursor::new(data);
let mut reader = SmartBuf::with_capacity(10, 2, cursor);

// Seek to absolute position
reader.seek(SeekFrom::Start(50)).unwrap();

// Seek relative to current position (forward)
reader.seek(SeekFrom::Current(10)).unwrap();

// Seek relative to current position (backward)
reader.seek(SeekFrom::Current(-5)).unwrap();

// Seek from end of file
reader.seek(SeekFrom::End(-10)).unwrap();

Reading Entire Files

use smartbuf::SmartBuf;
use std::io::Read;

let cursor = Cursor::new(vec![1, 2, 3, 4, 5]);
let mut reader = SmartBuf::new(cursor);

let mut contents = Vec::new();
reader.read_to_end(&mut contents).unwrap();

API Documentation

SmartBuf::new(reader: R) -> SmartBuf<R>

Creates a new SmartBuf with default settings:

• Buffer size: 8KB
• Queue length: 2

SmartBuf::with_capacity(bufsize: usize, queuelen: usize, reader: R) -> SmartBuf<R>

Creates a new SmartBuf with custom configuration:

• bufsize: Size of each buffer chunk in bytes
• queuelen: Number of buffers to keep in the pre-fetch queue (must be ≥ 1)

SmartBuf::position(&self) -> u64

Returns the current absolute position in the stream.

SmartBuf::buffer_size(&self) -> usize

Returns the configured buffer size.

Performance Considerations

• Buffer size: Larger buffers reduce the number of system calls but increase memory usage. A good default is 8KB–64KB.
• Queue length: More buffers in the queue allow for better pre-fetching, especially when reading sequentially. Values of 2–4 are usually sufficient.
• Seek optimization: Seeks within the currently buffered data are handled instantly. Seeks outside the buffer require synchronization with the background thread.

Examples

Reading with Seeking

use smartbuf::SmartBuf;
use std::io::{Read, Seek, SeekFrom, Cursor};

let data: Vec<u8> = (0..1000).collect();
let cursor = Cursor::new(data.clone());
let mut reader = SmartBuf::with_capacity(100, 2, cursor);

// Read first 50 bytes
let mut buf = vec![0; 50];
reader.read(&mut buf).unwrap();
assert_eq!(&buf, &data[0..50]);

// Seek to middle
reader.seek(SeekFrom::Start(500)).unwrap();
let mut buf = vec![0; 50];
reader.read(&mut buf).unwrap();
assert_eq!(&buf, &data[500..550]);

// Seek back
reader.seek(SeekFrom::Start(0)).unwrap();
let mut buf = vec![0; 50];
reader.read(&mut buf).unwrap();
assert_eq!(&buf, &data[0..50]);

Large File Processing

use smartbuf::SmartBuf;
use std::io::{Read, Seek, SeekFrom};
use std::fs::File;

let file = File::open("large_file.bin")?;
let mut reader = SmartBuf::with_capacity(64 * 1024, 4, file);

// Process file in chunks
let mut buffer = vec![0; 1024 * 1024]; // 1MB chunks
loop {
    match reader.read(&mut buffer)? {
        0 => break, // EOF
        n => {
            // Process buffer[..n]
            process_chunk(&buffer[..n]);
        }
    }
}

Requirements

• Rust 1.38.0 or later
• Dependencies: crossbeam-channel and crossbeam-utils for thread-safe communication

License

This project is licensed under the MIT License - see the LICENSE file for details.