Crates.io | turbo-json-checker |
lib.rs | turbo-json-checker |
version | 2.0.1 |
source | src |
created_at | 2021-08-17 15:03:37.871005 |
updated_at | 2021-08-20 20:45:38.294816 |
description | A pushdown automaton low memory JSON bytes stream checker returning the JSON root-type followed by its start and end index in the Reader |
homepage | https://github.com/irevoire/turbo-json-checker |
repository | https://github.com/irevoire/turbo-json-checker |
max_upload_size | |
id | 438500 |
size | 42,898 |
This is a fork of oxidized-json-checker that returns the json type and the json starting and ending index in the Reader.
This is a pure Rust version of the JSON_checker library.
This is a Pushdown Automaton that very quickly determines if a JSON text is syntactically correct. It could be used to filter inputs to a system, or to verify that the outputs of a system are syntactically correct.
You can use it with the std::io::Read
Rust trait to checked if a JSON is valid without having to keep it in memory.
I ran some tests against jq
to make sure the library when in the bounds.
I used a big JSON lines files (8.3GB) that I converted to JSON using jq -cs '.'
😜
You can find those Wikipedia articles on the benchmark repository of Paul Masurel's Tantivy.
jq type
How many times does jq
takes when it comes to checking and determining the type of a JSON document?
Probably too much, and also a little bit of memory: 12GB!
$ time cat ../wiki-articles.json | jq type
"array"
real 1m55.064s
user 1m37.335s
sys 0m21.935s
ojc
How many times does it takes to ojc
? Just a little bit less! It also consumes 0kb of memory.
$ time cat ../wiki-articles.json | ojc
Array
real 0m56.780s
user 0m47.487s
sys 0m12.628s
ojc
with SIMDHow many times does it takes to ojc
already? 56s, that can't be true, we are in 2020...
What about enabling some SIMD optimizations? Compile the binary with the nightly
feature and here we go!
$ cargo build --release --features nightly
$ time cat ../wiki-articles.json | ojc
Array
real 0m15.818s
user 0m10.892s
sys 0m10.721s