laps

Lexer and parser collections.

With laps, you can build lexers/parsers by just defining tokens/ASTs and deriving Tokenize/Parse trait for them.

Usage

Add laps to your project by running cargo add:

cargo add laps --features macros

Example

Implement a lexer for S-expression:

use laps::prelude::*;

#[token_kind]
#[derive(Debug, Tokenize)]
enum TokenKind {
  // This token will be skipped.
  #[skip(r"\s+")]
  _Skip,
  /// Parentheses.
  #[regex(r"[()]")]
  Paren(char),
  /// Atom.
  #[regex(r"[^\s()]+")]
  Atom(String),
  /// End-of-file.
  #[eof]
  Eof,
}

And the parser and ASTs (or actually CSTs):

type Token = laps::token::Token<TokenKind>;

token_ast! {
  macro Token<TokenKind> {
    [atom] => { kind: TokenKind::Atom(_), prompt: "atom" },
    [lpr] => { kind: TokenKind::Paren('(') },
    [rpr] => { kind: TokenKind::Paren(')') },
    [eof] => { kind: TokenKind::Eof },
  }
}

#[derive(Parse)]
#[token(Token)]
enum Statement {
  Elem(Elem),
  End(Token![eof]),
}

#[derive(Parse)]
#[token(Token)]
struct SExp(Token![lpr], Vec<Elem>, Token![rpr]);

#[derive(Parse)]
#[token(Token)]
enum Elem {
  Atom(Token![atom]),
  SExp(SExp),
}

The above implementation is very close in form to the corresponding EBNF representation of the S-expression:

Statement ::= Elem | EOF;
SExp      ::= "(" {Elem} ")";
Elem      ::= ATOM | SExp;

More Examples

See the examples directory, which contains the following examples:

• sexp: a S-expression parser.
• calc: a simple expression calculator.
• json: a simple JSON parser.
• clike: interpreter for a C-like programming language.

Accelerating Code Completion for IDEs

By default, Cargo does not enable optimizations for procedural macros, which may result in slower code completion if you are using laps to generate lexers. To avoid this, you can add the following configuration to Cargo.toml:

[profile.dev.build-override]
opt-level = 3

You can also try to manually enable/disable parallelization for lexer generation by adding:

#[derive(Tokenize)]
#[enable_par(true)] // or #[enable_par(false)]
enum TokenKind {
  // ...
}

The parallelization setting only affects compilation speed and has no effect at runtime, it's set automatically by laps by default.

Changelog

See CHANGELOG.md.

License

Copyright (C) 2022-2023 MaxXing. Licensed under either of Apache 2.0 or MIT at your option.