token-iter
| Crates.io | token-iter |
| lib.rs | token-iter |
| version | 0.1.0 |
| created_at | 2022-07-27 06:10:47.88658+00 |
| updated_at | 2022-07-27 06:10:47.88658+00 |
| description | A Rust crate that simplifies writing tokenizers |
| homepage | |
| repository | https://github.com/NicholasSterling/token-iter |
| max_upload_size | |
| id | 633750 |
| size | 45,140 |
Nicholas Sterling (NicholasSterling)
documentation
README
token-iter
A Rust library crate that facilitates writing tokenizers for textual languages.
Please read the API documentation here.
To use it, add this to your Cargo.toml:
[dependencies]
token-iter = "0.10.0"
And put this in your code:
use token_iter::*;
License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Contribution
Any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual-licensed as above, without any additional terms or conditions.
See CONTRIBUTING.md.