Round Pipers

A high-performance Rust library for streaming data processing with circular buffers and read-only pipes. Designed for simple developers who want to process data without dealing with complex Rust concepts.

Note there was definitely a lot of AI assistance to produce this. A human artisinally crafted by hand most of the core circular buffer code, and the streaming piping system (src/rw_pipe.rs), but a lot of the boiler plate where it's managing traits, and extracting common functionality around safe code has been guided by human hands but mostly written by various AI agent systems (goose + anthropic, goose + chat gpt, and claude code)

Circular Buffer Pipes (Streaming Data)

use round_pipers::Pipe;
use std::sync::Arc;

// Create a pipe for streaming f64 values
let pipe = Arc::new(Pipe::new("my_buffer", 1024, [])?);
let writer = pipe.clone().get_writer()?;
let reader = pipe.clone().get_reader();

// Write data
writer.write(100, |mut chunk, _state| {
    for i in 0..100 {
        chunk[i] = i as f64;
    }
})?;

// Read data with iterator - no lifetime management needed
for chunk_result in reader.iter_chunks(25, 25) {
    let chunk = chunk_result?;
    // Process chunk - automatically advances when dropped
    for &value in chunk.iter() {
        println!("Value: {}", value);
    }
}

Read-Only Pipes (Pre-populated Data)

use round_pipers::ReadOnlyPipe;

// Super simple API - just pass your slice
let data = vec![1.0, 2.0, 3.0, 4.0, 5.0];
let pipe = ReadOnlyPipe::new(&data, [])?;
let reader = pipe.get_reader();

// Same API as regular pipes - no learning curve
for chunk_result in reader.iter_chunks(2, 2) {
    let chunk = chunk_result?;
    for &value in chunk.iter() {
        println!("Value: {}", value);
    }
}

Key Features

• Zero-copy data processing with ndarray integration
• Thread-safe multiple readers, single writer model
• RAII memory management with automatic pointer advancement
• Unified API across all pipe types
• High-performance circular buffers using Linux memfd
• Multi-dimensional data support for complex shapes
• Sliding window processing with configurable read/consume patterns

Pipe Types

1. Circular Buffer Pipes: High-performance streaming with shared memory
2. Read-Only Pipes: Zero-copy access to existing data slices
3. Write-Only Pipes: Efficient buffer filling and streaming output

Examples

Run the included examples:

# Audio processing pipeline
cargo run --example audio_fft_pipeline

Requirements

• Linux (uses memfd_create and mmap)

Design Philosophy

Built for high-performance (zero-copy) streaming data with the goal of minimizing encounters with complex Rust concepts. The API prioritizes ease of use while maintaining zero-copy performance.