Development

maturin develop

Python bindings

>>> from fastok import PreTokenizer

>>> pre_tokenizer = PreTokenizer(model="gpt2")
>>> pre_tokenizer.pre_tokenize_str("My name is Alvaro and I live in Barcelona.")
['My', ' name', ' is', ' Alvaro', ' and', ' I', ' live', ' in', ' Barcelona', '.']
>>> pre_tokenizer.pre_tokenize(["My name is Alvaro and I live in Barcelona.", "I like pizza."])
[['My', ' name', ' is', ' Alvaro', ' and', ' I', ' live', ' in', ' Barcelona', '.'], ['I', ' like', ' pizza', '.']]