wetext-rs

A Rust implementation of WeText for text normalization in TTS (Text-to-Speech) applications.

Table of Contents

• wetext-rs
  • Table of Contents
  • Background
  • Features
  • Installation
  • FST Weight Files
    • Download Options
  • Usage
    • Basic Usage
    • With Configuration
    • Inverse Text Normalization (ITN)
    • Convenience Function
  • Configuration Options
  • Examples
    • Chinese Text Normalization
    • Chinese Inverse Text Normalization
    • English Text Normalization
    • Japanese Text Normalization
  • Dependencies
  • Compatibility with Python WeText
  • Development
    • Running Tests
    • Consistency Testing with Python WeText
    • Code Quality
  • Credits
  • License

Background

This project is a Rust port of the Python wetext library, which provides a lightweight runtime for WeTextProcessing without depending on Pynini. The primary motivation for creating this Rust implementation is to:

1. Integrate with the Candle ecosystem - Enable seamless integration with Rust-based ML frameworks like Candle, eliminating Python dependencies in production deployments
2. Improve performance - Leverage Rust's memory safety and zero-cost abstractions for faster text processing
3. Enable standalone deployment - Create a single binary that can be deployed without Python runtime

The original Python implementation uses kaldifst for FST operations. This Rust version uses rustfst, a pure Rust implementation of OpenFST, to achieve the same functionality.

Features

• Text Normalization (TN): Convert numbers, dates, currency to spoken form
  • "2024年1月15日" → "二零二四年一月十五日"
  • "$100" → "one hundred dollars"
• Inverse Text Normalization (ITN): Convert spoken form back to written form
  • "一百二十三" → "123"
• Multi-language support: Chinese (zh), English (en), Japanese (ja)
• English contractions expansion: "don't" → "do not"
• Various text preprocessing options:
  • Traditional to Simplified Chinese conversion
  • Full-width to half-width character conversion
  • Interjection removal
  • Punctuation removal
  • Erhua (儿化音) removal

Installation

Add to your Cargo.toml:

[dependencies]
wetext-rs = "0.1"

FST Weight Files

This library requires FST (Finite State Transducer) weight files for text normalization. The weight files can be downloaded from:

ModelScope: pengzhendong/wetext

Download the weight files and organize them in the following structure:

fsts/
├── traditional_to_simple.fst
├── full_to_half.fst
├── remove_interjections.fst
├── remove_puncts.fst
├── tag_oov.fst
├── en/
│   └── tn/
│       ├── tagger.fst
│       └── verbalizer.fst
├── zh/
│   ├── tn/
│   │   ├── tagger.fst
│   │   ├── verbalizer.fst
│   │   └── verbalizer_remove_erhua.fst
│   └── itn/
│       ├── tagger.fst
│       ├── tagger_enable_0_to_9.fst
│       └── verbalizer.fst
└── ja/
    ├── tn/
    │   ├── tagger.fst
    │   └── verbalizer.fst
    └── itn/
        ├── tagger.fst
        ├── tagger_enable_0_to_9.fst
        └── verbalizer.fst

Download Options

Option 1: ModelScope CLI

pip install modelscope
modelscope download --model pengzhendong/wetext --local_dir ./fsts

Option 2: Git LFS

git lfs install
git clone https://www.modelscope.cn/pengzhendong/wetext.git fsts

Usage

Basic Usage

use wetext_rs::{Normalizer, NormalizerConfig, Language, Operator};

// Create normalizer with default settings (Chinese TN, auto language detection)
let mut normalizer = Normalizer::with_defaults("path/to/fsts");

// Normalize text
let result = normalizer.normalize("2024年1月15日").unwrap();
println!("{}", result);  // 二零二四年一月十五日

With Configuration

use wetext_rs::{Normalizer, NormalizerConfig, Language, Operator};

// Configure for specific language and operation
let config = NormalizerConfig::new()
    .with_lang(Language::Zh)
    .with_operator(Operator::Tn)
    .with_fix_contractions(true)
    .with_traditional_to_simple(true);

let mut normalizer = Normalizer::new("path/to/fsts", config);
let result = normalizer.normalize("100元").unwrap();
println!("{}", result);  // 一百元

Inverse Text Normalization (ITN)

use wetext_rs::{Normalizer, NormalizerConfig, Language, Operator};

let config = NormalizerConfig::new()
    .with_lang(Language::Zh)
    .with_operator(Operator::Itn);

let mut normalizer = Normalizer::new("path/to/fsts", config);
let result = normalizer.normalize("一百二十三").unwrap();
println!("{}", result);  // 123

Convenience Function

use wetext_rs::normalize;

let result = normalize("path/to/fsts", "123").unwrap();
println!("{}", result);  // 幺二三

Configuration Options

Examples

Chinese Text Normalization

Chinese Inverse Text Normalization

English Text Normalization

Japanese Text Normalization

Dependencies

Compatibility with Python WeText

This Rust implementation is designed to be compatible with the Python wetext library. The core TN/ITN functionality produces identical results for the same inputs.

Differences from Python version:

Development

Running Tests

# Run all unit and integration tests
cargo test

# Run with verbose output
cargo test -- --nocapture

Consistency Testing with Python WeText

To verify that the Rust implementation produces identical results to the Python version:

cd tests
python3.13 -m venv venv
source venv/bin/activate
pip install wetext

python tests/generate_reference.py

cargo test test_compare_with_python -- --ignored --nocapture

✓ PASS: '123' (zh/tn) => '幺二三'
✓ PASS: '2024年1月15日' (zh/tn) => '二零二四年一月十五日'
...
Results: 20 passed, 0 failed

Code Quality

# Run clippy linter
cargo clippy -- -D warnings

# Format code
cargo fmt

# Check formatting
cargo fmt -- --check

Credits

• Original Python implementation: pengzhendong/wetext
• FST weight files: ModelScope - pengzhendong/wetext
• WeTextProcessing grammar: wenet-e2e/WeTextProcessing

License

This project is licensed under the Apache-2.0 License.