# Tokeneer

[![CI](https://github.com/InfiniTensor/tokeneer/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/InfiniTensor/tokeneer/actions)
[![Latest version](https://img.shields.io/crates/v/tokeneer.svg)](https://crates.io/crates/tokeneer)
[![Documentation](https://docs.rs/tokeneer/badge.svg)](https://docs.rs/tokeneer)
[![license](https://img.shields.io/github/license/InfiniTensor/tokeneer)](https://mit-license.org/)
![GitHub repo size](https://img.shields.io/github/repo-size/InfiniTensor/tokeneer)
![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/InfiniTensor/tokeneer)

[![GitHub Issues](https://img.shields.io/github/issues/InfiniTensor/tokeneer)](https://github.com/InfiniTensor/tokeneer/issues)
[![GitHub Pull Requests](https://img.shields.io/github/issues-pr/InfiniTensor/tokeneer)](https://github.com/InfiniTensor/tokeneer/pulls)
![GitHub contributors](https://img.shields.io/github/contributors/InfiniTensor/tokeneer)
![GitHub commit activity](https://img.shields.io/github/commit-activity/m/InfiniTensor/tokeneer)

Another tokenizer crate.