RunMat Lexer

This crate tokenizes MATLAB/Octave source code into a stream of tokens for the parser. It uses the logos library to define a fast, zero-copy DFA with a small amount of context via LexerExtras to handle MATLAB-specific ambiguities.

Design goals

• Correct tokenization for the full MATLAB language surface
• Minimal, explicit state for disambiguation (apostrophe transpose vs string, section markers, etc.)
• Compatibility with the rest of the toolchain (parser, HIR, interpreter, JIT)
• Predictable tokens: avoid over-encoding semantics at the lexing stage

Context-aware lexing

We track two pieces of context in LexerExtras:

• last_was_value: bool — true if the previous emitted token forms a value. Used to disambiguate ' as transpose vs string start.
• line_start: bool — true if we are at the beginning of a logical line. Used for %% section markers.

Tokens overview

• Keywords: function if elseif else for while break continue return end
• Additional keywords: switch case otherwise try catch global persistent true false
• OOP keywords: classdef properties methods events enumeration arguments
• Import: import
• Identifiers: [A-Za-z_][A-Za-z0-9_]*
• Numbers: integers and floats with optional exponents
• Strings:
  • Single-quoted character arrays: '...' with doubled quotes '' inside
  • Double-quoted string scalars: "..." with doubled quotes "" inside
• Operators and punctuation:
  • Arithmetic: + - * / \ ^
  • Element-wise: .* ./ .\ .^
  • Relational: == ~= < <= > >=
  • Logical: && || & | ~
  • Transpose: ' (contextual)
  • Colon: :
  • Dotted member access: .
  • Function handle/anonymous: @
  • Meta-class query: ? (e.g., ?MyClass)
  • Assignment and separators: = , ;
  • Grouping and containers: () [] {}
• Comments & layout:
  • Line comment: % to end of line
  • Section marker: %% at start of line
  • Block comment: %{ ... %} (non-nesting)
  • Line continuation: ... (skips remainder of physical line)
  • Newlines reset line_start

Notable disambiguations

• Apostrophe ':
  • If previous token was a value (identifier, number, ) ] }), emit Transpose
  • Otherwise, let the string regex capture a full single-quoted character array
• Section %%:
  • Only emitted when line_start == true; otherwise % starts a normal line comment
• Line continuation ...:
  • Emits Ellipsis and consumes the remainder of the physical line, including any % comment following it

Non-goals at lexing time

The lexer purposefully does not encode high-level semantics:

• Integer class names like int8/uint64 are identifiers
• Special variables like varargin/varargout/ans are identifiers
• OOP features (handle inheritance, method attributes) are parsed/handled later
• Command/function syntax duality is resolved in parsing/semantic phases

Tests

See tests/ for comprehensive coverage, organized by topic:

• lexer.rs: core tokens, operators, single-quoted strings, comments, ellipsis
• transpose.rs: detailed diagnostics and assertions for apostrophe (') transpose cases
• comments_continuation.rs: % line comments, %{...%} block comments, %% section markers, ... continuation
• operators.rs: logical and element-wise operators (e.g., .* ./ .\ .^ && || & | ~)
• namespaces.rs: import paths (including wildcard) and metaclass ?ClassName
• oop_tokens.rs: OOP keywords (classdef, properties, methods, events, enumeration, arguments) and function handles @
• strings_chars.rs: double-quoted string scalars and apostrophe disambiguation exercises
• tokens_basic.rs: identifiers, numbers, separators (; ,), and simple keyword smoke tests

All lexer tests pass when running the crate tests on their own.

Guidelines for extending the lexer

• Prefer adding new tokens only when lexical distinctions are required.
• When in doubt, keep ambiguous terms as identifiers and resolve in the parser.
• If you need context to disambiguate, add a boolean/flag in LexerExtras and use a Logos callback to Emit or Skip appropriately.
• Keep regular expressions simple (no look-around) and rely on token priority and callbacks for precedence and control.

Known compatibility notes

• Non-conjugate transpose .' is tokenized as Dot then Transpose. The parser should interpret this pair as the non-conjugating transpose.
• Block comments %{...%} are treated as non-nesting by design.
• Error-recovery is implemented to keep producing useful tokens after invalid input; in recovery mode double-quoted strings are recognized as a single Str token, while malformed single-quoted sequences may be split to allow downstream error reporting.

Remaining edges

• Apostrophe vs string: extreme adjacency cases across ... continuation and % comments are covered by tests; a few rare permutations may still be added as seeds (parser semantics unaffected).
• Block comments are intentionally non-nesting; any future change would be a parser/runtime decision, not lexing.
• Command-form is resolved in the parser; lexer's role is complete for milestone.

Crate integration

• This crate only produces tokens; it does not attempt to validate grammar.
• Downstream crates (runmat-parser, runmat-hir, runmat-ignition, runmat-turbine) are responsible for structure and semantics.