| Crates.io | wordcut-engine |
| lib.rs | wordcut-engine |
| version | 1.2.1 |
| created_at | 2017-12-27 15:16:42.013756+00 |
| updated_at | 2025-07-28 11:54:00.285433+00 |
| description | Word segmentation/breaking library |
| homepage | |
| repository | https://codeberg.org/mekong-lang/wordcut-engine |
| max_upload_size | |
| id | 44573 |
| size | 19,704,647 |
A word segmentation library in Rust for Southeast Asian and other languages. Currently, it supports Chinese, Japanese, Khmer, Myanmar (Burmese), Lao, Shan, Thai, and other space-delimited languages.
use wordcut_engine::load_dict;
use wordcut_engine::Wordcut;
use std::path::Path;
fn main() {
let dict_path = Path::new(concat!(
env!("CARGO_MANIFEST_DIR"),
"/dict.txt"
));
let dict = load_dict(dict_path).unwrap();
let wordcut = Wordcut::new(dict);
println!("{}", wordcut.put_delimiters("หมากินไก่", "|"));
}
Load this project as a dependency
:dep .
Import symbols
use wordcut_engine::load_dict;
use wordcut_engine::Wordcut;
use wordcut_engine::Dict;
use std::path::Path;
Initialize
let dict: Dict = load_dict("data/thai.txt").unwrap();
let wordcut = Wordcut::new(dict);
Running
let txt = "หมากินไก่";
wordcut.put_delimiters(txt, "|")
wordcut.build_path(txt, &txt.chars().collect::<Vec<_>>())
dbg!(wordcut.build_path(txt, &txt.chars().collect::<Vec<_>>()));
wordcut-engine has three steps:
Identifying clusters identify which substrings that must not be split.
For example,
[ก-ฮ]็
[ก-ฮ][่-๋]
[ก-ฮ][่-๋][ะาำ]
The above rules are wrapped with parentheses as shown below:
([ก-ฮ]็)
([ก-ฮ][่-๋])
([ก-ฮ][่-๋][ะาำ])
for example,
([ก-ฮ]็)|([ก-ฮ][่-๋])|([ก-ฮ][่-๋][ะาำ])
Building a DFA from the joined regular expression using regex-automata
Creating a directed acyclic graph (DAG) by adding edges using the DFA
Identifying clusters following a shortest path of a DAG from step above
Note: wordcut-engine does not allow a context sensitive rule, since it hurts the performance too much. Moreover, instead of longest matching, we use a DAG, and its shortest path to contraint cluster boundary by another cluster, therefore newmm-style context sensitive rules are not required.
In contrary to identifying clusters, identifying split-DAG edges identify what must be split. Split-DAG edge makers, wordcut-engine has three types of split-DAG edge maker, that are:
The dictionary-based maker traverses a prefix tree, which is particularly a trie in wordcut-engine and create an edge that matched word in the prefix tree. Rule-based maker uses regex-automata's Regex matcher built from split rules to find longest matched substrings, and add corresponding edges to the graph. wordcut-engine removes edges that break clusters. The example of split rules are shown below:
[\r\t\n ]+
[A-Za-z]+
[0-9]+
[๐-๙]+
[\(\)"'`\[\]{}\\/]
If there is no edge for each of character indice yet, a default maker create a edge that connected the known rightmost boundary.