EXPERIMENTAL/WIP

For now the lib.rs contains only a PoC

Test Coverage checks with Fault Injection

Fawlty can automatically inject faults at instrumented points and permute through all potential error paths of an test by restarting the test with the state from a former run. It can be used to provide information on whether a particular error is adequately treated in an application. Fawlty is only enabled in debug builds (debug_assertions). In release builds it emits non instrumented code.

Example

Code being instrumented with fawlty uses only the fawlty! macro which comes in 2 forms:

Statement fawlty!();

Creates a checkpoint on the callstack. At least any function that call fawlty instrumented functions should call this once at start.

Expression fawlty![good, bad..]

With a list of expressions that all have the same type as return type. This is the expression that can inject faults into the program. The first one must be the good case this is used when fawlty is disabled as well. There can be any number of bad expressions simulating various faults. These can be arbitrary complex (using braces). Fault injection will try these on after another in reverse order.

use fawlty::*;

fn main() {
    // enabling fawlty manually (FIXME: test driver)
    fawlty_enable();

    // any caller should put a checkpoint on the stack
    fawlty!();

    fn should_be_true() -> bool {
        // lets inject a false here
        fawlty![true, false]
    }

    // First call injects false (reverse order)
    assert_eq!(should_be_true(), false);
    // Any further call will return true
    assert_eq!(should_be_true(), true);
    assert_eq!(should_be_true(), true);

    fn zero_or_none() -> Option<u32> {
        fawlty![Some(0), None]
    }

    fn match_expr() -> &'static str {
        // injection can be done at any expression
        match fawlty![zero_or_none(), Some(1), None] {
            Some(0) => "zero",
            None => "none",
            _ => "oops",
        }
    }

    assert_eq!(match_expr(), "none");
    assert_eq!(match_expr(), "oops");
    assert_eq!(match_expr(), "none");
    assert_eq!(match_expr(), "zero");
}

Algorithm Details

Fawlty keeps track of the path leading to an instrumented expression by hashing the caller, the thread name, file, line and column together. Thus each unique way to reach an instrumented can be identified. A key:value state-store then keeps track of the states of all seen paths so far.

When a test runs with fawlty enabled it will proceed normally until an instrumented expression is hit. Then the state-store is consulted, a new instrumented path will be recorded with injecting the last (bad) expression of the given list. When the path was seen before then the next (in reverse) expression is selected for injection until if finally reaches the first good expression. Any bad injection disables further fault injections of this run. Thus fawlty (at this time) will only inject one single fault for each run and subsequent runs will permute through all possible single-fault code paths. In future this may become revised to handle multiple faults, whereat that may have much higher (exponential) costs to run tests.

For the time being fawlty tracks call paths with a shadow stack that needs instrumentation. This is chosen because its portable safe-rust to all platforms and very efficient. Future versions may (optionally) track call paths by BacktraceFrames when this become stabilized in rust.

Concurrency

Broken! currently failures will be injected in all threads in a racy way, this leads to non deterministic execution. A later version will do this in a more sorted way with only one error injected, permuting over all threads.