ellie_tokenizer
| Crates.io | ellie_tokenizer |
| lib.rs | ellie_tokenizer |
| version | 0.7.3 |
| created_at | 2023-07-30 20:08:20.51975+00 |
| updated_at | 2024-07-03 22:38:50.676306+00 |
| description | Tokenizer for ellie language |
| homepage | https://www.ellie-lang.org |
| repository | https://github.com/behemehal/Ellie-Language |
| max_upload_size | |
| id | 930221 |
| size | 376,647 |
Ahmetcan Aksu (ahmtcn123)
documentation
README
Ellie-Tokenizer
Tokenizer is the first stage of the pipeline. It takes a text and returns a list of items and their positions.