Table of Contents

• Code Product Library
• The Product Syntax
  • Product Scopes
  • Linear Scopes
  • Definitions
  • References
  • Escaping the $ character
• Examples
• Preprocessor like Macro Replacement
• EBNF Syntax

Code Product Library

This library implements an engine for expanding one TokenTree into another TokenTree by generating code productions.

The Product Syntax

The product syntax contains three elements described in detail below:

1. Product scopes for product definitions using curly braces:

   ${ ... }
   

2. Linear scopes that expand all defintions sequentially instead generating the product:

   $[ ... ]
   

3. The definitions using parentheses:

   $( ... )
   

4. References, either indexed or named:

   $0 $foo
   

Since we use Rust TokenTrees, the only requirement is that everything tokenizes. This means literals must be properly formatted, and all opening brackets must be closed.

All syntactic elements relevant to the product syntax start with the pound sign $.

Product Scopes

Allow local product definitions in $ followed by braces. These scopes are evaluated inside-out and expand the product of all definitions; named references that are not local will be forwarded to the parent scope.

${
    $(name: (one)(two))
    $(number: (1)(2))
    $name:$number ;
}

expands to:

one:1 ;
one:2 ;
two:1 ;
two:2 ;

Linear Scopes

Allow local definitions in $ followed by brackets. All definitions in a linear scope must have the same number of elements. These scopes are evaluated inside-out and expand the elements of all definitions together instead their product; named references that are not local will be forwarded to the parent scope.

$[
    $(name: (one)(two))
    $(number: (1)(2))
    $name:$number ;
]

expands to:

one:1 ;
two:2 ;

Definitions

A list of nested parentheses. The inner parentheses contain the produced Rust tokens. A product definition can be named or unnamed, hidden or visible. This leads to the three main forms (unnamed-hidden is not supported). For named definitions with only one item, the parentheses can be omitted.

Names can be used to reference the product at multiple places. This works for names from outer scopes as well. Unnamed definitions can only be referenced by their numeric position in the product list; these will be local and not reference outer scopes.

Hidden definitions will not be expanded at the place of their definition but can be referenced later. This is useful for defining a production first and using it multiple times. Visible definitions will be expanded at the place of their definition and can be referenced later as well.

• $((a)(b)) - unnamed, visible:
  Defining an unnamed product will implicitly leave a numeric reference to it in place. Thus, it will be expanded at the place of its definition. This is especially useful for defining a production that is used only once.
• $($name:(a)(b)) - named, visible:
  Defining a named production with a $ in front of its name will leave a reference to it in place. Thus, it will be expanded at the place of its definition.
• $(name:(a)(b)) - named, hidden:
  Defining a named production without the $ in front of its name will leave no reference to it. Thus, it will not be expanded at the place of its definition. This can be used when one wants to define a set of productions at the start of a scope which later become referenced.
• $($name:item) and $(name:item) - special form for single items:
  Single named non-parenthesized items can be written without the surrounding parentheses. Note: When the item needs to be parenthesized, one needs to put it in double parentheses.

The number of the product definitions starts with one; named definitions are included in the numbering.

Note 1:
Productions can only contain Rust tokens, not product definitions or references. This is intentionally chosen for simplicity for now.

Note 2:
Code expansion is the product of all defined products. Hidden products that are not referenced will still create the same code multiple times. This also applies to empty definitions:

$(unused: (1)()(3))
$(used: (a)(b))
foo($used)

will expand to:

foo(a)
foo(a)
foo(a)
foo(b)
foo(b)
foo(b)

References

References are used to reference an earlier defined production. They are written as $ followed by a number or the name of a named product definition. When named, this can refer to a name from an outer scope. Numbered references are only valid in the scope of the product definition they are defined; indexing starts at zero.

Unresolved references will lead to a compile error. Everything has to be defined before being used.

Escaping the $ character

The $ character is used to start product syntax elements. To use a literal $ character, just double it.

Examples

The following three examples will all expand to:

impl Trait<Foo> for MyType<Foo> {}
impl Trait<Bar> for MyType<Bar> {}
impl Trait<Baz> for MyType<Baz> {}

1. Using unnamed product definitions:

impl Trait<$((Foo)(Bar)(Baz))> for MyType<$1> {}

2. Using a named, hidden product definition:

$(T: (Foo)(Bar)(Baz))
impl Trait<$T> for MyType<$T> {}

3. Using a named, visible product definition:

impl Trait<$($T: (Foo)(Bar)(Baz))> for MyType<$T> {}

Preprocessor like Macro Replacement

When product definitions contain only one item, then code product expansion acts like a simple preprocessor, substituting defined products. Parentheses become optional then. For example:

$(T: Foo)
$(U: Bar)
impl Trait<$T> for MyType<$U> {}

will expand to:

impl Trait<Foo> for MyType<Bar> {}

EBNF Syntax

Here is an informal code product syntax in EBNF for reference. The actual implementation differs slightly depending on what ScopeMode is chosen.

start = { product_entity | rust } ;
rust = ?any_valid_rust_token? ;
product_entity = "$" ( scope | definition | reference ) ;
scope = product_scope | linear_scope ;
product_scope = "{" { product_entity | rust } "}" ;
linear_scope = "[" { product_entity | rust } "]" ;
definition = "(" [ "$" name ":" ] { product } ")" ;
product = "(" { rust } ")" ;
name = ?identifier? ;
reference = "#" ( ?number? | ?identifier? ) ;