ellie_tokenizer

Crates.ioellie_tokenizer
lib.rsellie_tokenizer
version0.7.3
sourcesrc
created_at2023-07-30 20:08:20.51975
updated_at2024-07-03 22:38:50.676306
descriptionTokenizer for ellie language
homepagehttps://www.ellie-lang.org
repositoryhttps://github.com/behemehal/Ellie-Language
max_upload_size
id930221
size376,647
Ahmetcan Aksu (ahmtcn123)

documentation

README

Ellie-Tokenizer

Tokenizer is the first stage of the pipeline. It takes a text and returns a list of items and their positions.

Commit count: 814

cargo fmt