palate_polyglot_tokenizer
| Crates.io | palate_polyglot_tokenizer |
| lib.rs | palate_polyglot_tokenizer |
| version | 0.2.1 |
| created_at | 2026-01-11 02:25:57.730327+00 |
| updated_at | 2026-01-11 02:25:57.730327+00 |
| description | A generic programming language tokenizer |
| homepage | https://github.com/casualjim/palate/tree/master/crates/palate_polyglot_tokenizer |
| repository | https://github.com/casualjim/palate/tree/master/crates/palate_polyglot_tokenizer |
| max_upload_size | |
| id | 2035110 |
| size | 42,615 |
Ivan Porto Carrero (casualjim)
documentation
README
Polyglot Tokenizer
A generic programming language tokenizer
Polyglot Tokenizer turns code into tokens, labelling each token as an Identifier, LineComment, BlockComment, Symbol, StringLiteral, or Number.
License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.