rten-text
| Crates.io | rten-text |
| lib.rs | rten-text |
| version | 0.14.0 |
| source | src |
| created_at | 2023-12-31 20:31:52.485095 |
| updated_at | 2024-10-27 17:52:05.841856 |
| description | Text tokenization and other ML pre/post-processing functions |
| homepage | https://github.com/robertknight/rten |
| repository | https://github.com/robertknight/rten |
| max_upload_size | |
| id | 1084992 |
| size | 93,440 |
Robert Knight (robertknight)
documentation
README
rten-text
Library containing text tokenization and related functionality, for preparing inputs and decoding outputs for text models (eg. BERT).
The functionality is a subset of that found in Hugging Face Tokenizers. It has less functionality, but also fewer dependencies, and none that require C/C++.