tokenmonster
| Crates.io | tokenmonster |
| lib.rs | tokenmonster |
| version | 0.1.0 |
| created_at | 2025-09-24 20:28:49.579313+00 |
| updated_at | 2025-09-24 20:28:49.579313+00 |
| description | Greedy tiktoken-like tokenizer with embedded vocabulary (cl100k-base approximator) |
| homepage | |
| repository | https://github.com/martinboros/native-vector-store |
| max_upload_size | |
| id | 1853790 |
| size | 1,274,228 |
Martin Boros (mboros1)
documentation
README
tokenmonster
Greedy tiktoken-like tokenizer with an embedded vocabulary, intended for fast, allocation-light tokenization.
Features
- Greedy tokenization compatible with common LLM vocabularies
- Zero-copy where possible; minimal allocations
- Optional tiny test vocabulary via the
tiny_vocabfeature
License: MIT