malk-lexer
| Crates.io | malk-lexer |
| lib.rs | malk-lexer |
| version | 0.1.1 |
| created_at | 2016-09-22 07:10:53.254379+00 |
| updated_at | 2016-09-24 05:51:08.997642+00 |
| description | A simple unicode lexer |
| homepage | https://github.com/canndrew/malk-lexer |
| repository | https://github.com/canndrew/malk-lexer |
| max_upload_size | |
| id | 6583 |
| size | 27,531 |
documentation
README
malk-lexer
A unicode lexer for use as a first-pass when writing a parser.
The main function exported by this library is lex which takes a &str and a
table of valid symbols and converts them to a token tree.
The kinds of token recognized by the lexer are:
- Idents: A string starting with a
XID_Startcharacter followed by a sequence ofXID_Continuecharacters. - Whitespace: Any sequence of whitespace characters.
- Brackets: Any bracket character, it's corresponding closing bracket and the tokens in-between returned as a sub-tree.
- Symbols: Any string that appears in the symbol table provided to
lex - Strings: A string enclosed with either
"or'and which may contain escaped characters.
Patches welcome!