💥 fastok

<div align="center">
  <h1>💥 fastok</h1>
  <strong>BPE in Rust with bindings to Python using PyO3</strong>
</div>

## Development

```
maturin develop
```

## Python bindings

```python
>>> from fastok import PreTokenizer

>>> pre_tokenizer = PreTokenizer(model="gpt2")
>>> pre_tokenizer.pre_tokenize_str("My name is Alvaro and I live in Barcelona.")
['My', ' name', ' is', ' Alvaro', ' and', ' I', ' live', ' in', ' Barcelona', '.']
>>> pre_tokenizer.pre_tokenize(["My name is Alvaro and I live in Barcelona.", "I like pizza."])
[['My', ' name', ' is', ' Alvaro', ' and', ' I', ' live', ' in', ' Barcelona', '.'], ['I', ' like', ' pizza', '.']]
```