# rust-covfix
[![Build Status](https://travis-ci.org/Kogia-sima/rust-covfix.svg?branch=master)](https://travis-ci.org/Kogia-sima/rust-covfix)
[![Build status](https://ci.appveyor.com/api/projects/status/xc9jauk9nah5webf/branch/master?svg=true)](https://ci.appveyor.com/project/Kogiasima/rust-covfix/branch/master)
[![codecov](https://codecov.io/gh/Kogia-sima/rust-covfix/branch/master/graph/badge.svg)](https://codecov.io/gh/Kogia-sima/rust-covfix)
[![Version](https://img.shields.io/crates/v/rust-covfix)](https://crates.io/crates/rust-covfix)
[![docs](https://docs.rs/rust-covfix/badge.svg)](https://docs.rs/rust-covfix)
![GitHub Release Date](https://img.shields.io/github/release-date/Kogia-sima/rust-covfix?label=last%20release)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/Kogia-sima/rust-covfix/blob/master/LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)

Rustc is known to report an incorrect coverage for some lines (https://stackoverflow.com/questions/32521800/why-does-kcov-calculate-incorrect-code-coverage-statistics-for-rust-programs).
`rust-covfix` will read coverage from the file generated by [grcov](https://github.com/mozilla/grcov/), fix it, then outputs the correct coverage.

Though only `lcov` format is supprted at current, Another formats is going to be supported in future releases.

## Features

- Compatible with the latest stable/beta/nightly Rust compiler
- Windows/OSX/Linux are all supprted
- Lightweight (small dependencies)
- Fast and safe (implemented in Rust language)
- `rust-covfix` is also available with Rust API ([Documentation](https://docs.rs/rust-covfix))
- Show summary of coverage difference.

### Optional features

Optional features are available with cargo's `--features` option. You can specify the features like:

```console
$ cargo install --no-default-features --features "cli lcov"
```

|Feature name|Description|Default?|
|:--:|--|:--:|
|cli|Command Line Interface. This feature is required to build `rust-covfix` executable.|yes|
|lcov|Make LcovParser available|yes|
|noinline|Avoid adding `#cfg[inline]` attribute on function. (deprecated)|no|
|backtrace|Dump backtrace information on every time the error has occured.|no|

## Install

Download the latest release from [GitHub Release Page](https://github.com/Kogia-sima/rust-covfix/releases).

You can also install via `cargo` command.

```console
$ cargo install rust-covfix
```

## How to generate correct code coverage from Rust program?

#### 1. Avoid inlining the functions (optinal)

It seems that the current version of rustc (1.42) will automatically inline the function that is only called from one place.
This behaviour causes incorrect coverage for your tests.

To avoid this, you have to add `#[inline(never)]` attributes for their functions manually. I recommend defining a new feature flag `coverage` in your crate. In Cargo.toml, append the following lines.

```Cargo.toml
[features]
coverage = []
```

And then, add the attribute like `#[cfg(feature = "coverage", inline(never))]` to the functions which is called from just one location.

```rust
#[cfg(feature = "coverage", inline(never))]
fn foo() {
  // ...
}
```

This will avoid inlining the functions only when you enable `coverage` feature flag.

#### 2. Compile your crate with `-Zprofile` option

In order to generate code coverage with rustc, you must specify `-Zprofile` option.
This option is currently (1.42) unstable and only available from nightly toolchain.

Also, some other flags will be required to generate **correct** coverage. Here is my recommend.

```console
$ export CARGO_INCREMENTAL=0
$ export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"
```

Then, compile your crate and run tests.

```console
$ cargo test --features coverage
```

#### 3. aggregate code coverage data

Now there are code coverage data in `target/debug/deps` directory.
The next step is aggregating them and convert the format so that `rust-covfix` can read coverage data.

I highly recommend to use [grcov](https://github.com/mozilla/grcov/) to aggregate them. This project is developed by mozilla team and supports the latest Rust toolchains.

Install latest `grcov` and run the following commands from your project root directory.

```console
zip -0 ccov.zip `find . \( -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" \) -print`
./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info
```

where `YOUR_PROJECT_NAME` is your crate name specified in `Cargo.toml`.

#### 4. Fix coverage data using rust-covfix

Now rust-covfix can read coverage from `lcov.info`.

```console
$ rust-covfix -o lcov_correct.info lcov.info
```

This command will write a **correct** coverage into `lcov_correct.info`. You can upload them into codecov.io, or generate HTML summary using `genhtml`.

#### Use rust-covfix on Travis CI

Here is an example script to use `rust-covfix` on Travis CI environment.

```sh
#!/bin/bash
# ci/script.sh

set -ex

if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
  # Setup grcov
  wget https://github.com/mozilla/grcov/releases/download/v0.5.7/grcov-linux-x86_64.tar.bz2
  tar xvf grcov-linux-x86_64.tar.bz2

  # Setup environmental variables
  export CARGO_INCREMENTAL=0
  export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"
fi

# Compile and run tests
cargo test --all-features

if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
  # collect coverages
  zip -0 ccov.zip `find . \( -name "rust_covfix*.gc*" -o -name "test-*.gc*" \) -print`
  ./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info

  # fix coverage using rust-covfix
  rust-covfix lcov.info -o lcov.info

  # upload coverage to codecov
  bash <(curl -s https://codecov.io/bash) -f lcov.info
fi
```

Then, call this script from `travis.yml`

```yaml
language: rust

matrix:
  include:
    - os: linux
      rust: nightly

script:
  - bash ci/script.sh
```

## Why is this project developped as a standalone package?

There are 3 reasons for this.

1. A grcov collaborator is aware of rustc reporting incorrect coverage (https://github.com/mozilla/grcov/issues/249#issuecomment-465154051). I'm trying to compare original Rust code and transformed LLVM IR, but still don't fully understand why Rustc report incorrect coverage. The `rust-covfix` was originally developped as a temporal workaround for this problem until I understand when and where rustc reports incorrect coverage. It is not completely unraveled yet.

2. `grcov` is just a coverage collection tool, not a coverage generator. It only aggregates the raw coverage data, and convert the format. It does not read the source code, nor manipulate any coverage data. Also, although grcov is targetting many programming languages such as C/C++, Nim and Rust, `rust-covfix` only supports Rust. 

3. `rust-covfix` is targetting not only grcov, but also [cargo-kcov](https://github.com/kennytm/cargo-kcov) and [Tarpaulin](https://github.com/xd009642/tarpaulin) in the future. Actually, `Tarpaulin` can now generate the `lcov.info`. It means `rust-covfix` is already able to fix the coverage data generated by `Tarpaulin`. I'm working on [feature/cobertura branch](https://github.com/Kogia-sima/rust-covfix/tree/feature/cobertura) to support `cargo-kcov` too. The final objective is to support these 3 tools.

## How is the incorrect line coverage detected

`rust_covfix` fixes the coverage information using some rules. You can pass `--rules` option to specify which rules are used to fix coverages.

### Rules

#### close

closing brackets, line of `else` block will be ignored.

```rust
if a > 0 {
    b = a
} else {  // <-- marked as "not executable"
    b = -a
};  // <-- marked as "not executable"
```

#### test

module block named `test` or `tests` which has attribute `cfg(test)` will be ignored.

all functions with `#[test]` attribute will be also ignored.

```rust
#[cfg(test)]
mod tests {  // <-- removed from coverage
    fn util() { ... }  // <-- removed from coverage

    #[test]
    fn test_hoge() { ... }  // <-- removed from coverage
}
```

#### loop

Fix rust internal bugs that loop branches are not correctly passed.

```rust
for i in 0..10 {  // <-- fix branch coverage information
    println!("{}", i);
}
```

#### derive

structs with `derive(...)` attribute will be ignored

```rust
#[derive(Clone, Debug)]  // <-- removed from coverage
struct Point {  // <-- removed from coverage
  x: f64,  // <-- removed from coverage
  y: f64  // <-- removed from coverage
}  // <-- removed from coverage
```

#### comment

ignore coverage based on comment marker

```rust
fn main() {
    let a = 1 + 2; // cov:ignore-line

    // cov:begin-ignore-branch
    println!("Hello");
    println!("world!");
    // cov:end-ignore-branch

    // cov: begin-ignore-line
    if a > 2 {
        println!("a is large!");
    } else if a == 0 {
        println!("a is small!");
    }
    // cov: end-ignore-line

    // cov:begin-ignore
    println!("a = {}", a);
    // cov:end-ignore

    println!("finish."); // cov:ignore-branch

    return (); // cov:ignore
}
```

## Roadmap

- Support `cobertura.xml` file. (WIP)
- Add option for uploading the correct coverages to coveralls.
- Use syntax tree generated using [syn](https://crates.io/crates/syn) crate.
- Performance improvement

## Author

👤 **Kogia-sima**

* Twitter: [@Kogia\_sima](https://twitter.com/Kogia\_sima)
* Github: [@Kogia-sima](https://github.com/Kogia-sima)

## 🤝 Contributing

Contributions, issues and feature requests are welcome!

Feel free to check [issues page](https://github.com/Kogia-sima/rust-covfix/issues). 

## Show your support

Give a ⭐️ if this project helped you!

## 📝 License

Copyright © 2019 [Kogia-sima](https://github.com/Kogia-sima).

This project is [MIT](https://github.com/Kogia-sima/rust-covfix/blob/master/LICENSE) licensed.

***
_This README was generated with ❤️ by [readme-md-generator](https://github.com/kefranabg/readme-md-generator)_