sentencepiece-model
| Crates.io | sentencepiece-model |
| lib.rs | sentencepiece-model |
| version | 0.1.4 |
| created_at | 2023-11-18 14:31:03.094167+00 |
| updated_at | 2024-10-08 17:19:28.796631+00 |
| description | SentencePiece model parser generated from the SentencePiece protobuf definition |
| homepage | |
| repository | https://github.com/Systemcluster/sentencepiece-model |
| max_upload_size | |
| id | 1040527 |
| size | 23,729 |
Chris (Systemcluster)
documentation
README
sentencepiece-model
SentencePiece model parser generated from the SentencePiece protobuf definition.
use sentencepiece_model::SentencePieceModel;
let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer()?.unk_id(), 2);
Usage
[dependencies]
sentencepiece-model = "0.1"
sentencepiece-model uses prost-build and protox to generate Rust code from the SentencePiece protobuf definition at build time. protoc is not required.