# wordpieces

This crate provides a subword tokenizer. A subword tokenizer splits a
token into several pieces, so-called *word pieces*.  Word pieces were
popularized by and used in the
[BERT](https://arxiv.org/abs/1810.04805) natural language encoder.