Crates.io | ellie_tokenizer |
lib.rs | ellie_tokenizer |
version | 0.7.3 |
source | src |
created_at | 2023-07-30 20:08:20.51975 |
updated_at | 2024-07-03 22:38:50.676306 |
description | Tokenizer for ellie language |
homepage | https://www.ellie-lang.org |
repository | https://github.com/behemehal/Ellie-Language |
max_upload_size | |
id | 930221 |
size | 376,647 |
Tokenizer is the first stage of the pipeline. It takes a text and returns a list of items and their positions.