text-tokenizer
| Crates.io | text-tokenizer |
| lib.rs | text-tokenizer |
| version | 0.6.12 |
| created_at | 2023-12-20 11:59:26.645572+00 |
| updated_at | 2025-07-24 09:39:48.07809+00 |
| description | Custom text tokenizer |
| homepage | https://github.com/merl-twin/text-tokenizer |
| repository | https://github.com/merl-twin/text-tokenizer |
| max_upload_size | |
| id | 1075234 |
| size | 524,569 |
(merl-twin)
documentation
README
Custom text tokenizer