tocken

Crates.iotocken
lib.rstocken
version0.1.0
created_at2025-01-02 04:27:43.253717+00
updated_at2025-01-02 04:27:43.253717+00
descriptionClustering algorithms.
homepage
repositoryhttps://github.com/kemingy/tocken
max_upload_size
id1501266
size43,991
Keming (kemingy)

documentation

https://docs.rs/tocken

README

Tocken

CI crates.io docs.rs

Tokenizer implemented in Rust.

This tokenizer is based on Lucene's EnglishAnalyzer.

Usage

  • as a library: check the main.rs file and docs.
  • as a CLI:
    • cargo r -r --help
    • cargo r -r -- -i wiki.txt -o wiki_tocken_f10.json -f 10
Commit count: 8

cargo fmt