text-tokenizer
| Crates.io | text-tokenizer |
| lib.rs | text-tokenizer |
| version | 0.5.10 |
| source | src |
| created_at | 2023-12-20 11:59:26.645572 |
| updated_at | 2024-11-22 08:35:25.859514 |
| description | Custom text tokenizer |
| homepage | https://github.com/merl-twin/text-tokenizer |
| repository | https://github.com/merl-twin/text-tokenizer |
| max_upload_size | |
| id | 1075234 |
| size | 487,719 |
(merl-twin)
documentation
README
Custom text tokenizer