bert_tokenizer
| Crates.io | bert_tokenizer |
| lib.rs | bert_tokenizer |
| version | 0.1.3 |
| source | src |
| created_at | 2023-02-14 14:21:10.429488 |
| updated_at | 2023-02-15 07:05:14.943461 |
| description | This crate is a Rust port of Google's BERT WordPiece tokenizer. |
| homepage | |
| repository | https://github.com/yigit353/bert_tokenizer |
| max_upload_size | |
| id | 784927 |
| size | 470,365 |
Yiğit Bekir Kaya (yigit353)
documentation
README
bert_tokenizer
The crate provides the port of the original BERT tokenizer from the Google BERT repository.
License
MIT license. See LICENSE file for full license.