# skipif

`skipif` provides an attribute macro called `skip_if`. This macro allows the
user to specify that a rust test case should be skipped if a specified
condition is met. `skip_if` currently supports the following conditions:

* `missing_env(VAR1, VAR2, ...)`. If the specified environment variable(s) is not set on the
  `cargo test` process then the test case should be skipped.

This is useful for tests that integrate with external systems such as
Kubernetes, databases, cloud APIs, etc.

When one or more specified conditions are met then the macro rewrites the test
case name to append `_SKIPPED` to avoid the misleading appearance of a test
case passing as might normally be the case for a test requiring an environment
variable to specify an external integration config.

## Motivation

Programming language test runners often offer some way to programmatically mark
a test case as skipped at runtime. A good example of this is Golang, where test
cases can call the [`testing.T.SkipNow` method][gotestskip] to specify that the
current running test case should be marked as skipped. In the Golang case, it's
up to the programmer to determine whether a test case should be skipped for any
reason and this is how they do it.

In Rust, we have the builtin `#[ignore]` attribute macro, but it lacks any
mechanism for programmatically determining at runtime whether or not to skip a
given test. Instead, the best recourse we have is either a macro like `skip_if`
OR returning early indicating success or failure. For example:

```rust
#[test]
fn my_supercool_test_succeeds() {
  if std::env::var("DATABASE_URL").is_err() {
    // variable is unset. oh well, i guess this test passes
    return;
  }
}

#[test]
fn my_supercool_test_fails() {
  // variable is unset. oh well, i guess this test fails
  assert!(std::env::var("DATABASE_URL").is_ok());
}
```

## How `skip_if` Works


This is what a test case utilizing `skip_if` looks like:

```rust
#[skipif::skip_if(missing_env(DATABASE_URL))]
#[test]
fn my_supercool_test() -> std::result::Result<(), ()> {
  assert!(false);
}
```

During the macro expansion phase of the `cargo test` compilation, the macro
checks for the presence of an environment variable named `DATABASE_URL`. If
found, it expands to essentially the same test fn as shown. If the variable
isn't found, the following happens:

* the test name gets `_SKIPPED` appended
* the test body is removed such that the test case trivially passes
* the test fn output signature is set to `-> ()`

So we end up with a test case that looks more like:

```rust
#[test]
fn my_supercool_test_SKIPPED() {}
```

The advantage, compared to the `my_supercool_test_succeeds` example shown
above is that instead of ending up with the following in our test suite output:

```
test my_supercool_test ... ok
```

we get:

```
test my_supercool_test_SKIPPED ... ok
```

The `_SKIPPED` at the end gives developers runnign the test case the
information they need to know that the test case didn't merely pass but was
SKIPPED.

Many folks might prefer the behavior in the `my_supercool_test_fails` example
above and that's fine. `skip_if` is intended for people who don't want that
behavior.

## Gotchas

### Attribute macro order

Attribute macro order matters here. The `[skipif::skip_if(...)]` macro must
precede any `[test]`-like macro (eg `[tokio::test]`, `[sqlx::test]`). If
the `skip_if` macro doesn't get chance to rename the test function then a
`[#test]`-like macro will capture the wrong function name to be executed in
test `main`.

### Recompilation

This macro leads to different behaviors at test run time based on the
conditions present during test compile time. But the `cargo` compilation
step doesn't automatically recognize, for example with the `missing_env`
contidion) that an environment variable relevant to tests has changed.

Taking the `my_supercool_test` example above as an example, imagine the
following sequence of events:

1. `DATABASE_URL` is unset and you run `cargo test`.
1. The `cargo test` process compiles the test binary.
   * During compilation, the macro is expanded and we end up with
    `my_supercool_test_SKIPPED`.
1. The `cargo test` process executes the compiled binary.
1. The test case `my_supercool_test_SKIPPED` trivially passes.
1. You realize you need to set `DATABASE_URL` so you do so without changing any
   code. You run `cargo test` again.
1. The `cargo test` process executes the compiled binary.
1. The test case `my_supercool_test_SKIPPED` trivially passes.

But wait, why didn't the second invocation of `cargo test` re-compile the test
binary? Because `cargo` (or `rustc`, I don't know) caches build output and only
re-compiles if one of the compilation inputs changes. In the case of
`my_supercool_test` and the `missing_env` condition, arbitrary environment
variables are not considered to be compilation inputs.

However, not all is lost! There is a workaround to fix this in the form of
[`cargo` build scripts][builtscript]. Specifically, you can use the
[`cargo:rerun-if-env-changed=<SUPERCOOL_ENVVAR>`][rerunifenvvar] instruction
from a build script to tell `cargo` to consider `<SUPERCOOL_ENVVAR>` as a
compilation input. Here is an example build script that would do the trick for
`my_supercool_test`:

```rust
fn main() {
  println!("cargo:rerun-if-env-changed=DATABASE_URL")
}
```

Just drop that into `build.rs` right next to `Cargo.toml` and you would get
test cases that recompile whenever the `DATABASE_URL` value changes. The
downside of this approach is that everything else in your crate would also get
rebuilt ¯\_(ツ)_/¯ whenever the variable changes.

## Similar projects

* [needs_env_var](https://github.com/HerrMuellerluedenscheid/needs_env_var)


[gotestskip]: https://pkg.go.dev/testing#T.Skip
[buildscript]: https://doc.rust-lang.org/cargo/reference/build-scripts.html
[rerunifenvvar]: https://doc.rust-lang.org/cargo/reference/build-scripts.html#rerun-if-env-changed