# malk-lexer

A unicode lexer for use as a first-pass when writing a parser.

The main function exported by this library is `lex` which takes a `&str` and a
table of valid symbols and converts them to a token tree.

The kinds of token recognized by the lexer are:
 * **Idents**: A string starting with a `XID_Start` character followed by a
   sequence of `XID_Continue` characters.
 * **Whitespace**: Any sequence of whitespace characters.
 * **Brackets**: Any bracket character, it's corresponding closing bracket and
   the tokens in-between returned as a sub-tree.
 * **Symbols**: Any string that appears in the symbol table provided to `lex`
 * **Strings**: A string enclosed with either `"` or `'` and which may contain
   escaped characters.

Patches welcome!