text-tokenizer
| Crates.io | text-tokenizer |
| lib.rs | text-tokenizer |
| version | 0.6.16 |
| created_at | 2023-12-20 11:59:26.645572+00 |
| updated_at | 2025-10-30 12:21:33.850935+00 |
| description | Custom text tokenizer |
| homepage | https://github.com/merl-twin/text-tokenizer |
| repository | https://github.com/merl-twin/text-tokenizer |
| max_upload_size | |
| id | 1075234 |
| size | 536,777 |
(merl-twin)
documentation
README
Custom text tokenizer