Cargo Regression

Collect test task from input files, execute them and compare results with golden.

Usage

cargo install cargo-regression

Or you can download the latest/examples.zip, and call ./regression instead of cargo regression.

Then set your test files like ./demo, and

cargo regression ./demo

The tests will be exectued in ./tmp in default, change the directory by --workdir. All CLI arguments here:

Set Extension(s)

cargo-regression will collect all files that match extensions as test tasks, you can set extensions in two ways:

• By commmand arguments

cargo regression ./demo --extensions py sh

• By xx/__all__.toml

# override for all sub-dir
extensions = ["py", "sh"]

Other Config

The full configs demo is demo/full.toml. Except extensions can only be defined in xx/__all__.toml, the other configs can be defined in both xx/__all__.toml and xxx.toml. There are three types of configs:

• xx/__all__.toml: Affect all task for current and all sub directories
• xxx.toml: Only affect for input task file xxx.xx
• Argument: Is equivalent to set it in the most top /__all__.toml

Variable Table

There are a few keywords that will be replaced into its values, for all configs.

Extend Config

In default the configs will be override after you define them in xxx.toml. But for args, envs, and extern-files, you can extend them base on the super's configs. See test-extend.toml

Workflow

Overall:

1. Collect all tasks
2. Parallel test each task, parallelism controlled by permit

For each task:

1. Preprocess Cmds
2. Execute Cmds
3. Postprocess Cmds
4. Assertion
5. Collect & Report

Advanced Features

Test Filter

Only test specified tasks.

# No filter
cargo regression ./demo
# Only include demo/trybuild/* and ./demo/test-sh/*
cargo regression ./demo --include demo/trybuild/* ./demo/test-sh/*
# Exclude demo/trybuild/*
cargo regression ./demo --exclude demo/trybuild/*
# Combined filter
cargo regression ./demo --include demo/trybuild/* --exclude demo/trybuild/compile-ok.rs

Schedule Parallelism

permits and permit are virtual resource costs, you can define permits in arguments (default=1), and define permit in task toml config file (default=0). See test-premit

cargo regression ./demo --include demo/test-premit/* --permits 1
cargo regression ./demo --include demo/test-premit/* --permits 2

Preprocess & Postprocess

You can define one or more pre/post-paration action(s).

[[preprocess]]
cmd = "cargo"
args = ["build", "--release", "--examples"] # default: []
workdir = "{{rootdir}}/.." # default: the task's workdir
[[postprocess]]
cmd = "cargo"
args = ["build", "--release", "--examples"] # default: []
workdir = "{{rootdir}}/.." # default: the task's workdir

assertion

exit-code

Assert the exit code, default is 0. See test-exit.toml

[assert]
exit-code = 1

equal

The output file should equal to the golden. See compile-fail.toml

[[assert.golden]]
file = "{{name}}.stderr"
# The task's stder' should equal to __golden__/{{name}}.stderr
equal = true

match

Match pattern and assert the number (count) of it. See test-match.toml

[[assert.golden]]
# match pattern from task stdout
file = "{{name}}.stdout"
match = [
  # regular expression match
  { pattern = 'f.*o', count = 4 },
  # should contain word "fo" at least once
  { pattern = '\bfo\b', count-at-least = 1 },
  # should contain word "fo" at most once
  { pattern = '\bfo0\b', count-at-most = 1 },
]

value

Capture float number and assert the value (count) of it. The epsilon is assert tolerance, if the epsilon is not defined, default epsilon is 1e-10 See test-value.toml

[[assert.golden]]
# match float value from task stdout
file = "{{name}}.stdout"
value = [
  # match 'f.*o' and then match a float, assert it should within 4±0.01
  { pattern-before = 'f.*o', value = 4.0, epsilon = 0.01 },
  # match a float and then match 'after\b', assert it should > 1-0.0000000001
  { pattern-after = 'after\b', value-at-least = 1 },
  # math the float between 'f.*o' and 'after\b', assert it should within 4±0.0000000001
  { pattern-before = 'f.*o', pattern-after = 'after\b', value = 4.0 }
]

custom

Use external custom assert script to compare output and golden. The script's first/second arguments are output/golden file path, and should return 0 when assertion is success. envs will be passed as environment variables. See test-custom.toml

# For each file matches {{name}}.*.out,
# do
# cd workdir && {{rootdir}}/cmp1.sh {{name}}.xxx.out __golden__/{{name}}.xxx.out
# and
# cd workdir && {{rootdir}}/cmp2.sh {{name}}.xxx.out __golden__/{{name}}.xxx.out (with env)
[[assert.golden]]
file = "{{name}}.*.out"
custom = [
  { cmd = "{{rootdir}}/cmp1.sh" },
  { cmd = "{{rootdir}}/cmp2.sh", envs = {ABS_ERR="1e-10"} },
]

Use its library

use cargo_regression::{test, Args, TestExitCode};

#[tokio::main]
async fn main() -> TestExitCode {
  // Get arguments from CLI
  let args = Args::parse_from(std::env::args_os());
  // Or set fixed arguemnts
  let args = Args::new("./demo");
  args.test().await
}

See more in ./examples

Reminder: For fixed argument, the include and exclude variables should be all files matched by yourself, e.g., using ["path/A", "path/B"] rather than ["path/*"].