📚 Csv Lib v1.0.6

A high-performance, zero-copy CSV reader for Rust, optimized for extremely fast parsing using:

• 🔥[ New ] Supports multithread complete Row process with a clousure,and a editable shared object
• Memory-mapped files (memmap2)
• SIMD acceleration (AVX2 on x86_64, NEON on aarch64)
• memchr3 fallback for broad CPU compatibility
• Configurable line breaks, delimiters, and string escaping
• Low memory overhead, even on massive datasets

✨ Features

• 🚀 Memory-mapped CSV reading (no BufReader overhead)
• 🚀 SIMD acceleration (AVX2, NEON) if available
• 🚀 Fallback to memchr3 for full CPU compatibility
• 🚀 Per-row zero-copy parsing
• 🚀 Per-field parsing using efficient iterators
• 🚀 Support for custom delimiters and string escaping
• 🚀 Support for column type mapping or auto-detection
• 🚀 Optional FFI export for C, C++, Python, C#, and Java between other options
• 🚀 Safe cursor management
• 🚀 UTF-8, Windows1252 and custom encoding support

📊 Performance Benchmark – csv_lib Rust Library vs other libraries

We benchmarked the performance of processing 1,000,000,000 CSV rows using several popular Rust libraries. Each result is averaged over 3 independent runs. The test is get &str of each line. In this library we NOT decode the fields, untile you ask for it un the struct field get value, we just get the bytes of the line. For this example we decoded the full row, which is not so efficient.

🔥 Why Choose csv_lib?

• ✅ Faster than csv and csv-core even in single-threaded mode.
• 🧵 True parallelism: multi-core support with chunked processing and low synchronization overhead.
• 🚀 Built on memmap2 and SIMD (AVX2 / NEON) for fast parsing directly from memory.
• 🧠 Configurable closures for per-line processing and custom splitting logic.
• 📁 Capable of handling files larger than 13 GB without memory bottlenecks.

📦 Best Suited For

• Massive CSV file ingestion (ETL, logs, data lakes)
• High-performance backend pipelines
• Lightweight embedded CLI applications with zero-copy data access

💡 csv_lib is optimized for sequential, chunked, and parallel processing using memory-mapped I/O and customizable parsing logic.

⚙️ Installation

Put in your terminal

cargo add csv_lib

If you also want FFI support:

cargo add csv_lib --features ffi

In your project folder, at the same level src create a .cargo/config.toml file with the following content:

[build]
rustflags = ["-C", "target-cpu=native"]

Usage:

We use Row and Field struct, to handle the navigation in the document.

• For a full example project check examples folder.
• Check the macros here

One Core Example:

pub fn main(){
    //Create Config
    let cfg = CsvConfig::new(
        b',',
        0u8,
        b'\n',
        Encoding::Windows1252,
        false
    );
    //Open the file
    let f = match CsvReaderWithMap::open("data.csv", &cfg) {
        Ok(f) => f,
        Err(e) => panic!("{}", e)
    };
    // We extract different' s country's of the dataset :
    // For example:
    //Create a Hash Acumulator
    let mut cities :HashSet<String>= HashSet::with_capacity(195);
    //Iter over rows
    while let Some(mut row) = f.next_raw() {
        //Extract Field index 6 starting on 0
        let city = row.get_index(6 );
        // Decode bytes as &str
        let name = city.get_utf8_as_str();
        //Check and accumulate
        if !cities.contains(name){
            cities.insert(name.to_string());
        }
    }
}

Multicore Example:

pub fn main(){
    //Create Config
    let cfg = CsvConfig::new(
        b',',
        0u8,
        b'\n',
        Encoding::Windows1252,
        false
    );
    //Open the file
    let f = match CsvReaderWithMap::open("data.csv", &cfg) {
        Ok(f) => f,
        Err(e) => panic!("{}", e)
    };

    //Get Slice Reference
    let data = f.get_slice();
    //Create a shared counter
    let shared = Shared::<i32>::default();
    //Create de clousere executed on each thread (the ARC Mutex type must be the same as Shared)
    let closure = |_: &mut RowParallel<'_>, target: Arc<Mutex<i32>>| {
        //Do some stuff
        // ...
        //Access editable variable.(Use after process due it blocks). Omit this lock uf you can.
        let mut lock = target.lock().unwrap();
        *lock += 1;
    };
    //Execute parallel process
    parallel_processing_csv(
        data,
        b'\n',
        b',',
        0u8,
        false,
        closure,
        shared.arc(),
    );
    println!("Iterated Lines: {}", shared.lock())
}

📊 CSV Parsing Benchmark Results between different functions of the library

• Full line decoding: If you decode only needed fields performance will be better.
• File Size: data_1000000000.txt (approx. 14GB)
• Test Metric: Time in milliseconds (ms) — lower is better

Changelog

Check it here

📈 Performance Tips

If you are going to test performance of the library, do it in release mode. It have a huge difference due the trash lines of code cargo generates in debug profile, and the time of process is awful

• Use force_memcach3 = false to take advantage of SIMD (AVX2 or NEON).
• Match your delimiter, line_break, and string_separator properly to the file format.
• Prefer UTF-8 / Windows-1252 encodings for maximum parsing speed.
• Process fields immediately without copying them if possible (&[u8] slices).

🔗 Useful Links

The reached performance was possible due this 3 crates

• Rust memmap2 crate
• memchr crate (SIMD optimized)
• num_cpus

🏆 Author

Made by Ignacio Pérez Panizza 🇺🇾 🧉