tocken

Crates.io	tocken
lib.rs	tocken
version	0.1.0
created_at	2025-01-02 04:27:43.253717+00
updated_at	2025-01-02 04:27:43.253717+00
description	Clustering algorithms.
homepage
repository	https://github.com/kemingy/tocken
max_upload_size
id	1501266
size	43,991

Keming (kemingy)

documentation

https://docs.rs/tocken

Tokenizer implemented in Rust.

This tokenizer is based on Lucene's EnglishAnalyzer.

as a library: check the main.rs file and docs.
as a CLI:
- cargo r -r --help
- cargo r -r -- -i wiki.txt -o wiki_tocken_f10.json -f 10

Commit count: 8