# Introduction

The wumpus hunter finds flaky tests: test cases that sometimes fail,
but sometimes work. It runs the automated test suite for some software
repeatedly, and logs the results.

# Acceptance criteria

Overall note, we need a git repository for the tests. Rather than
using an existing one available on the Internet, we create a new one
for each scenario. This avoids the acceptance test suite depending on
a remote repository working.

## Helper script to create cargo test project

~~~{#init-repo.sh .sh .file}
#!/bin/bash

set -xeuo pipefail

name="$1"
cargo init "$name"
cd "$name"
git config --local user.email subplot@example.com
git config --local user.name Subplot
git add .
git commit -m init
~~~

## Smoke test with successes

_Requirement:_ The wumpus hunter runs repeatedly a configuration where
every test suite run succeeds, but ends after the specified number of
runs.

_Justification:_ This allows us to verify that at least something
works.

~~~scenario
given an installed wumpus hunter
given file smoke.yaml
given a directory logdir
given file init-repo.sh
when I run bash init-repo.sh proj
when I run wumpus-hunter run --repeats 10 --logs logdir smoke.yaml
then file logdir/stats.txt contains "SUCCESS"
then file logdir/stats.txt doesn't contain "FAILURE"
then there are 10 successful runs and 0 failed runs in logdir
~~~

~~~{#smoke.yaml .file .yaml}
description: test repo
repository_url: proj
git_ref: main
command: true
~~~

## Smoke test with failures

_Requirement:_ The wumpus hunter runs repeatedly a configuration where
every test suite run fails, but ends after the specified number of
runs.

_Justification:_ This allows us to verify that at least something
works. In combination with the smoke test for successes, this scenario
verifies the wumpus hunter doesn't just create the requested number of
successful logs.

~~~scenario
given an installed wumpus hunter
given file smoke-fail.yaml
given a directory logdir
given file init-repo.sh
when I run bash init-repo.sh proj
when I run wumpus-hunter run --repeats 10 --logs logdir smoke-fail.yaml
then file logdir/stats.txt contains "FAILURE"
then file logdir/stats.txt doesn't contain "SUCCESS"
then there are 0 successful runs and 10 failed runs in logdir
~~~

~~~{#smoke-fail.yaml .file .yaml}
description: test repo
repository_url: proj
git_ref: main
command: false
~~~

## Clone repository

_Requirement:_ The wumpus hunter must clone a git repository, or
update an already cloned checkout.

_Justification:_ Without this, there is no point.

We create a repository using `cargo` and `git`.

~~~scenario
given an installed wumpus hunter
given file simple-repo.yaml
given a directory logdir
given file init-repo.sh
when I run bash init-repo.sh simple
when I run wumpus-hunter run --repeats 10 --logs logdir simple-repo.yaml
then there are 10 successful runs and 0 failed runs in logdir
~~~

~~~{#simple-repo.yaml .file .yaml}
description: test repo
repository_url: simple
git_ref: main
command: |
  grep '^name = "simple"' Cargo.toml
~~~
## Get updates

_Requirement:_ The wumpus hunter fetches updates to the repository it
runs tests on.

_Justification:_ The wumpus hunter needs to test latest changes.

~~~scenario
given an installed wumpus hunter
given file updated-repo.yaml
given a directory logdir
given file init-repo.sh
when I run bash init-repo.sh simple

when I run wumpus-hunter run --repeats 10 --logs logdir updated-repo.yaml
then there are 0 successful runs and 10 failed runs in logdir

when I run sed -i 's/name = "simple"/name = "updated-name"/' simple/Cargo.toml
when I run bash -c 'cd simple && git commit -am rename'
when I run mv logdir logdir.old
given a directory logdir

when I run wumpus-hunter run --repeats 5 --logs logdir updated-repo.yaml
then there are 5 successful runs and 0 failed runs in logdir
~~~

~~~{#updated-repo.yaml .file .yaml}
description: test repo
repository_url: simple
git_ref: main
command: |
  grep -Fx 'name = "updated-name"' Cargo.toml
~~~

## Test run takes too long

_Requirement:_ The wumpus hunter fails a test run that takes too long.

_Justification:_ If a test never finishes, the wumpus hunter would
stop and that's not acceptable.

~~~scenario
given an installed wumpus hunter
given file slow.yaml
given a directory logdir
given file init-repo.sh
when I run bash init-repo.sh simple

when I run wumpus-hunter run --repeats 2 --logs logdir slow.yaml --timeout 1
then there are 0 successful runs and 2 failed runs in logdir
~~~

~~~{#slow.yaml .file .yaml}
description: test repo
repository_url: simple
git_ref: main
command: |
  sleep 5
~~~

## Checks for temporary files from test

_Requirement:_ The wumpus hunter should optionally check that the test
suite doesn't leave temporary files behind, after the run has
finished.

_Justification:_ Test suites that don't clean up after themselves can
fill up the system they're run on with junk.

~~~scenario
given an installed wumpus hunter
given file temp.yaml
given a directory logdir
given file init-repo.sh
when I run bash init-repo.sh simple

when I run wumpus-hunter run --repeats 2 --logs logdir temp.yaml --check-tempdir
then there are 0 successful runs and 2 failed runs in logdir
~~~

~~~{#temp.yaml .file .yaml}
description: test repo
repository_url: simple
git_ref: main
command: |
  touch "$TMPDIR/$(date -Is)"
~~~