canbench

canbench is a tool for benchmarking canisters on the Internet Computer.

Background

Canister smart contracts on the Internet Computer consume compute and memory resources. Given that resources are finite, there are bounds in place when canisters execute a message (transaction):

1. Instructions: a monotonically increasing counter that's corelated with the amount of computation and memory accesses.
2. Dirty Pages: the number of memory pages that are written to.

A single message execution must stay within the allowed bounds, otherwise it's terminated. canbench provides developers the tools and insights to understand how their code is using instructions and memory. Support for reporting dirty pages will be available once there's a way to retrieve dirty page information from the IC.

Use Cases

• Understanding how a canister consumes instructions, heap memory, and stable memory.
• Detecting performance regressions locally or on CI.
• Analyzing where performance bottlenecks are present.

Features

• Metrics that are relevant

  Typically benchmarking tools run a benchmark multiple times and return the average time. On the Internet Computer, where instrumentation is deterministic, this approach is neither ideal nor insightful. Instead, canbench reports the number of instructions consumed, as well as changes to both heap and stable memories.

• Easy detection of regressions

  canbench allows you to persist the benchmarking results in your canister's repository. Storing the benchmarking results allows canbench to determine how the performance has changed relative to the past to detect regressions.

• Generous instruction limit

  While messages on the Internet Computer are bound to a few billion instructions, canbench can run benchmarks that are up to 10 trillion instructions, giving you the freedom to write resource-intensive benchmarks as needed.

• Language Agnostic

  canbench can, in theory, benchmark canisters written in any language. Initially support for only Rust exists, but support for additional languages can easily be introduced.

Installation

cargo install canbench

Quickstart (Rust)

See the crate's documentation.

Github CI Support

canbench can be included in Github CI to automatically detect performance changes. Have a look at the workflows in this repository for working examples. You'll need the following:

1. Scripts for executing the benchmark, which can be found in the scripts directory.
2. A workflow for canbench to post a comment with the benchmarking results. See canbench-post-comment.yml.
3. A job for uploading the PR number to the artifacts. This is necessary for step #2 to work correctly. See upload-pr-number in ci.yml.
4. The benchmarking job in CI. See benchmark-fibonacci-example in ci.yml.

Once you have the CI workflow set up, the job will pass if there are no significant performance changes detected and fail otherwise. A comment is added to the PR to show the results. See this PR for an example.