Goldenscript

A Rust testing framework loosely based on Cockroach Labs' datadriven framework for Go. It combines several testing techniques that make it easy and efficient to write and update test cases:

• Golden master testing (aka characterization testing or historical oracle)
• Data-driven testing (aka table-driven testing or parameterized testing)
• Keyword-driven testing

A goldenscript is a plain text file that contains a set of arbitrary input commands and their expected text output, separated by ---:

command
---
output

command argument
command key=value
---
output

The commands are executed by a provided Runner. The expected output is usually not written by hand, but instead generated by running tests with the environment variable UPDATE_GOLDENFILES=1:

$ UPDATE_GOLDENFILES=1 cargo test

The files are then verified by inspection and checked in to version control. Tests will fail with a diff if they don't match the expected output.

This approach is particularly useful when testing complex stateful systems, such as database operations, network protocols, or language parsing. It can be tedious and labor-intensive to write and assert such cases by hand, so scripting and recording these interactions often yields much better test coverage at a fraction of the cost.

Internally, the goldenfile crate is used to manage golden files.

Documentation

See the crate documentation which has more information on syntax and features.

Examples

For real-world examples, see e.g.:

• toyDB Raft: distributed consensus cluster.
• toyDB MVCC: ACID transactions.
• goldenscript parser: Goldenscript uses itself to test its parser and runner.

Below is a basic example, testing the Rust standard library's BTreeMap.

# Tests the Rust standard library BTreeMap.

# Get and range returns nothing for an empty map.
get foo
range
---
get → None

# Inserting keys out of order will return them in order. Silence the insert
# output with ().
(insert b=2 a=1 c=3)
range
---
a=1
b=2
c=3

# Getting a key returns its value.
get b
---
get → Some("2")

# Bounded scans, where the end is exclusive.
range b
---
b=2
c=3

range a c
---
a=1
b=2

# An end bound less than the start bound panics. Expect the failure with !.
!range b a
---
Panic: range start is greater than range end in BTreeMap

# Replacing a key updates the value and returns the old one.
insert b=foo
get b
---
insert → Some("2")
get → Some("foo")

The corresponding runner for this script:

#[derive(Default)]
struct BTreeMapRunner {
    map: std::collections::BTreeMap<String, String>,
}

impl goldenscript::Runner for BTreeMapRunner {
    fn run(&mut self, command: &goldenscript::Command) -> Result<String, Box<dyn Error>> {
        let mut output = String::new();
        match command.name.as_str() {
            // get KEY: fetches the value of the given key, or None if it does not exist.
            "get" => {
                let mut args = command.consume_args();
                let key = &args.next_pos().ok_or("key not given")?.value;
                args.reject_rest()?;
                let value = self.map.get(key);
                writeln!(output, "get → {value:?}")?;
            }

            // insert KEY=VALUE...: inserts the given key/value pairs, returning the old value.
            "insert" => {
                let mut args = command.consume_args();
                for arg in args.rest_key() {
                    let old = self.map.insert(arg.key.clone().unwrap(), arg.value.clone());
                    writeln!(output, "insert → {old:?}")?;
                }
                args.reject_rest()?;
            }

            // range [FROM] [TO]: iterates over the key/value pairs in the range from..to.
            "range" => {
                use std::ops::Bound::*;
                let mut args = command.consume_args();
                let from = args.next_pos().map(|a| Included(a.value.clone())).unwrap_or(Unbounded);
                let to = args.next_pos().map(|a| Excluded(a.value.clone())).unwrap_or(Unbounded);
                args.reject_rest()?;
                for (key, value) in self.map.range((from, to)) {
                    writeln!(output, "{key}={value}")?;
                }
            }

            name => return Err(format!("invalid command {name}").into()),
        };
        Ok(output)
    }
}


#[test]
fn btreemap() {
    goldenscript::run(&mut BTreeMapRunner::default(), "btreemap").expect("goldenscript failed")
}