Crates.io | bulloak |
lib.rs | bulloak |
version | 0.7.4 |
source | src |
created_at | 2023-08-05 18:56:32.600521 |
updated_at | 2024-07-10 13:46:12.829482 |
description | A Solidity test generator based on the Branching Tree Technique. |
homepage | https://github.com/alexfertel/bulloak |
repository | https://github.com/alexfertel/bulloak |
max_upload_size | |
id | 936716 |
size | 453,473 |
A Solidity test generator based on the Branching Tree Technique.
[!WARNING] Note that
bulloak
is still0.*.*
, so breaking changes may occur at any time. If you must depend onbulloak
, we recommend pinning to a specific version, i.e.,=0.y.z
.
cargo install bulloak
The following VSCode extensions are not essential but they are recommended for a better user experience:
.tree
filesbulloak
implements two commands:
bulloak scaffold
bulloak check
Say you have a foo.tree
file with the following contents:
FooTest
└── When stuff is called // Comments are supported.
└── When a condition is met
└── It should revert.
└── Because we shouldn't allow it.
You can use bulloak scaffold
to generate a Solidity contract containing
modifiers and tests that match the spec described in foo.tree
. The following
will be printed to stdout
:
// $ bulloak scaffold foo.tree
// SPDX-License-Identifier: UNLICENSED
pragma solidity 0.8.0;
contract FooTest {
modifier whenStuffIsCalled() {
_;
}
function test_RevertWhen_AConditionIsMet() external whenStuffIsCalled {
// It should revert.
// Because we shouldn't allow it.
}
}
You can use the -w
option to write the generated contracts to the file system.
Say we have a bunch of .tree
files in the current working directory. If we run
the following:
$ bulloak scaffold -w ./**/*.tree
bulloak
will create a .t.sol
file per .tree
file and write the generated
contents to it.
If a .t.sol
file's title matches a .tree
in the same directory, then
bulloak
will skip writing to that file. However, you may override this
behaviour with the -f
flag. This will force bulloak
to overwrite the
contents of the file.
$ bulloak scaffold -wf ./**/*.tree
Note all tests are showing as passing when their body is empty. To prevent this,
you can use the -S
(or --vm-skip
) option to add a vm.skip(true);
at the
beginning of each test function. This option will also add an import for
forge-std's Test.sol
and all test contracts will inherit from it.
You can skip emitting the modifiers by passing the -m
(or --skip--modifiers
)
option. This way, the generated files will only include the test functions.
You can use bulloak check
to make sure that your Solidity files match your
spec. For example, any missing tests will be reported to you.
Say you have the following spec:
HashPairTest
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.
And a matching Solidity file:
pragma solidity 0.8.0;
contract HashPairTest {
function test_ShouldNeverRevert() external {
// It should never revert.
}
function test_WhenFirstArgIsSmallerThanSecondArg() external {
// It should match the result of `keccak256(abi.encodePacked(a,b))`.
}
}
This Solidity file is missing the tests for the branch
When first arg is bigger than second arg
, which would be reported after
running bulloak check tests/scaffold/basic.tree
, like so:
warn: function "test_WhenFirstArgIsBiggerThanSecondArg" is missing in .sol
+ fix: run `bulloak check --fix tests/scaffold/basic.tree`
--> tests/scaffold/basic.tree:5
warn: 1 check failed (run `bulloak check --fix <.tree files>` to apply 1 fix)
As you can see in the above message, bulloak
can fix the issue automatically.
If we run the command with the --stdout
flag, the output is:
--> tests/scaffold/basic.t.sol
pragma solidity 0.8.0;
contract HashPairTest {
function test_ShouldNeverRevert() external {
// It should never revert.
}
function test_WhenFirstArgIsSmallerThanSecondArg() external {
// It should match the result of `keccak256(abi.encodePacked(a,b))`.
}
function test_WhenFirstArgIsBiggerThanSecondArg() external {
// It should match the result of `keccak256(abi.encodePacked(b,a))`.
}
}
<--
success: 1 issue fixed.
Running the command without the --stdout
flag will overwrite the contents of
the solidity file with the fixes applied. Note that not all issues can be
automatically fixed, and bulloak's output will reflect that.
warn: 13 checks failed (run `bulloak check --fix <.tree files>` to apply 11 fixes)
You can skip checking that the modifiers are present by passing the -m
(or --skip--modifiers
) option. This way, bulloak
will not warn when a
modifier is missing from the generated file.
The following rules are currently implemented:
.tree
& .t.sol
.bulloak scaffold
, is present in
the Solidity file.bulloak scaffold
,
matches the spec order.
bulloak scaffold
are checked. This means that any number of
extra functions, modifiers, etc. can be added to the file.Another feature of bulloak
is reporting errors in your input trees.
For example, say you have a buggy foo.tree
file, which is missing a └
character. Running bulloak scaffold foo.tree
would report the error like this:
•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
bulloak error: unexpected `when` keyword
── when the id references a null stream
^^^^
--- (line 2, column 4) ---
file: foo.tree
bulloak scaffold
scaffolds Solidity test files based on .tree
specifications
that follow the
Branching Tree Technique.
Currently, there is on-going discussion on how to handle different edge-cases to better empower the Solidity community. This section is a description of the current implementation of the compiler.
when/given
branches of a tree.it
branches of a tree.Each tree
file should describe at least one function under test. Trees follow
these rules:
bulloak
expects you to use ├
and └
characters to denote branches.when
or given
, it is a condition.
when
and given
are interchangeable.it
, it is an action.
it
is the same as It
and IT
.//
is a comment and will be stripped from the
output.Take the following Solidity function:
function hashPair(bytes32 a, bytes32 b) private pure returns (bytes32) {
return a < b ? hash(a, b) : hash(b, a);
}
A reasonable spec for the above function would be:
HashPairTest
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.
There is a top-level action that will generate a test to check the function invariant that it should never revert.
Then, we have the two possible preconditions: a < b
and a >= b
. Both
branches end in an action that will make bulloak scaffold
generate the
respective test.
Note the following things:
bulloak
also
supports actions with sibling conditions, but this might get removed in a
future version per this
discussion.Suppose you have additional Solidity functions that you want to test in the same
test contract, say Utils
within utils.t.sol
:
function min(uint256 a, uint256 b) private pure returns (uint256) {
return a < b ? a : b;
}
function max(uint256 a, uint256 b) private pure returns (uint256) {
return a > b ? a : b;
}
The full spec for all the above functions would be:
Utils::hashPair
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.
Utils::min
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the value of `a`.
└── When first arg is bigger than second arg
└── It should match the value of `b`.
Utils::max
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the value of `b`.
└── When first arg is bigger than second arg
└── It should match the value of `a`.
Note the following things:
bulloak
to error.
This violation is not currently fixable with bulloak check --fix
so will
need to be manually corrected.test_MinShouldNeverRevert
).There are a few things to keep in mind about the scaffolded Solidity test:
.tree
but with a .t.sol
extension. E.g. test.tree
would correspond to test.t.sol
..tree
file.Please refer to CONTRIBUTING.md.
These are the current steps taken to publish bulloak
:
git cliff -o CHANGELOG.md
. This step includes setting the proper header for
the latest tag.cargo publish --dry-run
to make sure that everything looks good.cargo publish
.This project has been possible thanks to the support of:
This project is licensed under either of: