# pretok

Pretok is a pre-tokenizer for C-like syntaxes.  Pretok simplifies subsequent tokenizers by handling line and block comments, whitespace and strings.  Pretok operates as an iterator over an input string of UTF-8 code points.

Given an input string, pretok does the following.
* Implements the iterator trait where ``next()`` returns a sequence of ``Option<Pretoken>`` structures.
* Filters ``// line comments`` from the input string.
* Filters ``/* block comments */`` from the input string
* Returns ``"quoted strings with \"escapes\""`` as a single ``Pretoken``.
* Skips whitespace characters.
* After above filters, returns ``Pretokens`` usually delineated by whitespace.
* Returns the line number and byte offset of each pretoken


## Examples

Whitespace typically separates pretokens and is stripped outside of quoted strings.

    let mut pretok = PreTokenizer::new("Hello World!");
    assert!(pretok.next() == Some(Pretoken{s:"Hello", line:1, offset:0}));
    assert!(pretok.next() == Some(Pretoken{s:"World!", line:1, offset:6}));
    assert!(pretok.next() == None);

Comments are stripped and may also delineate pretokens.

    let mut pretok = PreTokenizer::new("x/*y*/z");
    assert!(pretok.next() == Some(Pretoken{s:"x", line:1, offset:0}));
    assert!(pretok.next() == Some(Pretoken{s:"z", line:1, offset:6}));
    assert!(pretok.next() == None);

    let mut pretok = PreTokenizer::new("x\ny//z");
    assert!(pretok.next() == Some(Pretoken{s:"x", line:1, offset:0}));
    assert!(pretok.next() == Some(Pretoken{s:"y", line:2, offset:2}));
    assert!(pretok.next() == None);

Quoted strings are a single pretoken.

    let mut pretok = PreTokenizer::new("Hello \"W o r l d!\"");
    assert!(pretok.next() == Some(Pretoken{s:"Hello", line:1, offset:0}));
    assert!(pretok.next() == Some(Pretoken{s:"\"W o r l d!\"", line:1, offset:6}));
    assert!(pretok.next() == None);

Quoted strings create a single pretoken separate from the surrounding pretoken(s).

    let mut pretok = PreTokenizer::new("x+\"h e l l o\"+z");
    assert!(pretok.next() == Some(Pretoken{s:"x+", line:1, offset:0}));
    assert!(pretok.next() == Some(Pretoken{s:"\"h e l l o\"", line:1, offset:2}));
    assert!(pretok.next() == Some(Pretoken{s:"+z", line:1, offset:13}));
    assert!(pretok.next() == None);


## Unit Testing
Pretok supports unit tests.

    cargo test

## Fuzz Testing
Pretok supports fuzz tests.  Fuzz testing starts from a corpus of random inputs and then further randomizes those inputs to try to cause crashes and hangs.  At the time of writing (Rust 1.46.0), fuzz testing **required the nightly build**.

To run fuzz tests:

    cargo +nightly fuzz run fuzz_target_1

Fuzz tests run until stopped with Ctrl-C.  In my experience, fuzz tests will catch a problem almost immediately or not at all.

Cargo fuzz uses LLVM's libFuzzer internally, which provides a vast array of runtime options.  To see thh options using the nightly compiler build:

    cargo +nightly fuzz run fuzz_target_1 -- -help=1

For example, setting a smaller 5 second timeout for hangs:

    cargo +nightly fuzz run fuzz_target_1 -- -timeout=5