logreduce-tokenizer
| Crates.io | logreduce-tokenizer |
| lib.rs | logreduce-tokenizer |
| version | 0.1.0 |
| source | src |
| created_at | 2022-02-10 20:22:16.352423 |
| updated_at | 2022-02-10 20:22:16.352423 |
| description | A tokenizer function for the logreduce project. |
| homepage | |
| repository | https://github.com/logreduce/logreduce-tokenizer |
| max_upload_size | |
| id | 530432 |
| size | 64,714 |
Tristan de Cacqueray (TristanCacqueray)
documentation
https://docs.rs/logreduce-tokenizer/
README
logreduce-tokenizer
Build:
python setup.py build
export PYTHONPATH=$(pwd)/build/lib
Bench:
python benches/bench.py
Test perf:
RUSTFLAGS="-C target-cpu=native" cargo build --release
Build CLI:
RUSTFLAGS="-C target-cpu=native" cargo build --example logreduce-tokenizer-cli --release