# Contributing

Here are specific details that are useful when you want to contribute to the BPE crates.
Make sure to read the repository's [contribution guidelines][contributing] as well.

## Project structure

This project has a slightly unusual structure to resolve some dependency issues.

- This directory contains `bpe`, the BPE code itself.
- A sibling directory contains `bpe-openai`, which exposes tokenizers for OpenAI token sets, and depends on `bpe`.
- Tests are located in the `tests` subdirectory, and benchmarks in the `benchmarks` subdirectory. Both of these are separate crates so they can depend on `bpe-openai` without causing a cyclic dependency.

Only the `bpe` and `bpe-openai` crates are meant to be published. The other ones are for development use only.

## Running benchmarks

Change the working directory to the `benchmarks` directory:

```sh
cd benchmarks
```

Run the benchmark as follows (required [cargo-criterion](https://crates.io/crates/cargo-criterion) installed):

```sh
cargo criterion
```

(Using `cargo bench` ignores the settings in `criterion.toml`!)
Open the full report which should be located in `target/criterion/reports/index.html`.

Update the figures in this repo as follows (requires `rsvg-convert` from `librsvg` installed):

```sh
script/copy-results
```

[contributing]: ../../CONTRIBUTING.md