justlex-rs

Configurable lexer with no dependencies in Rust.

The entire implementation lives in a single .rs file. This means that you can copy-paste it in your project and modify it as you like, keeping your dependency tree clean. You can easily add custom Punctuation or Keywords by modifying the file directly.

Check out the examples in the examples directory.

• Author: Giovanni Santini
• Mail: giovanni.santini@proton.me
• License: MIT

Example

// Some input string
let example_input = String::from(
    "#include <stdio.h>
    if (a == 17*2 + 35) { // single line comment
        /* multi
         * line
         * comment
         */
        return b;
    }\n");

// Setup the lexer
let mut lexer = JustLex::default();
lexer.set_input(example_input);

// Iterate over the tokens
for token in lexer {
    let token = token;
    println!("Received token {:?}", token);
    if token == Token::None {
        break;
    }
}

Will output:

Received token Punct(Hash)
Received token Symbol("include")
Received token Punct(Less)
Received token Symbol("stdio")
Received token Punct(Dot)
Received token Symbol("h")
Received token Punct(Greater)
...

Usage

Run an example:

cargo run --example basic_usage

Run the tests:

cargo test

Run lint:

cargo clippy