generic_tokenizer
| Crates.io | generic_tokenizer |
| lib.rs | generic_tokenizer |
| version | 0.1.0 |
| created_at | 2024-10-27 16:46:53.660012+00 |
| updated_at | 2024-10-27 16:46:53.660012+00 |
| description | A generic tokenizer that tracks line and column numbers as it goes. |
| homepage | |
| repository | https://github.com/mr-adult/JFC |
| max_upload_size | |
| id | 1424767 |
| size | 10,751 |
Adam Fortune (mr-adult)
documentation
https://docs.rs/generic_tokenizer/latest/generic_tokenizer/
README
Generic Tokenizer
This crate provides a generic tokenizer that includes line and column numbers. This is useful for things like higher-level tokenizers, parsers, and language servers that need to track line and column numbers as they go.