sentencepiece-model

Crates.iosentencepiece-model
lib.rssentencepiece-model
version0.1.0
sourcesrc
created_at2023-11-18 14:31:03.094167
updated_at2023-11-18 14:31:03.094167
descriptionSentencepiece model parser
homepage
repositoryhttps://github.com/Systemcluster/sentencepiece-model
max_upload_size
id1040527
size21,820
Chris (Systemcluster)

documentation

README

sentencepiece-model

Crates.io Docs.rs

SentencePiece model parser generated from the SentencePiece protobuf definition.

use sentencepiece_model::SentencePieceModel;

let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer()?.unk_id(), 2);
Commit count: 4

cargo fmt