large-json-array

Stream large JSON arrays in Rust using an efficient iterator-based approach.

🚀 Overview

Serde's default behavior loads entire JSON documents into memory. When dealing with very large JSON arrays, this becomes inefficient or outright impossible. This crate provides a custom JsonStream<R: Read> iterator that streams and parses objects from a JSON array one-by-one without allocating the entire array in memory.

🔧 Features

• Memory-efficient streaming of large JSON arrays
• Works with any Read source (e.g., files, network streams)
• Integrates with serde_json::Value
• Robust bracket tracking and string-state handling

🛠 Example

use std::fs::File;
use std::io::BufReader;
use large_json_array::json_stream::JsonStream;
use serde_json::Value;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let file = File::open("huge.json")?;
    let reader = BufReader::new(file);
    let stream = JsonStream::new(reader);

    for value in stream {
        match value {
            Ok(json_value) => println!("{:?}", json_value),
            Err(e) => eprintln!("Error: {}", e),
        }
    }

    Ok(())
}

You can also run the examples, provided in the /examples directory:

Generate a large json file first:

    cargo run --example generate_large_json  

Then you can find all the Johns in the json (with --release flag it will run faster):

    cargo run --release --example find_johns users_5.0_gb.json    

⚙️ How It Works

The core struct is:

pub struct JsonStream<R: Read> {
    ...
}

It implements Iterator<Item = Result<Value, JsonError>> and maintains:

• A buffer for reading chunks
• A character accumulator for partial objects
• Bracket and string context state

Parsing logic handles array delimiters, quoted strings, escape sequences, and ensures objects are emitted only when fully formed.

📥 Installation

In your Cargo.toml:

[dependencies]
large-json-array = "0.1.0"

❗ Limitations

• Assumes well-formed JSON arrays of objects (e.g., [ {...}, {...}, ... ])
• Strings must be UTF-8 encoded
• Not suitable for deeply malformed JSON (for now)

📄 License

This project is licensed under the terms of the MIT license.

For more advanced use cases or improvements (e.g., error recovery, typed deserialization, or multi-threaded parsing), contributions and ideas are welcome!