ragit-korean
| Crates.io | ragit-korean |
| lib.rs | ragit-korean |
| version | 0.4.3 |
| created_at | 2024-12-30 14:17:57.361309+00 |
| updated_at | 2025-09-15 15:33:47.895117+00 |
| description | korean tokenizer for ragit |
| homepage | |
| repository | https://github.com/baehyunsol/ragit |
| max_upload_size | |
| id | 1499190 |
| size | 65,959 |
(baehyunsol)
documentation
https://docs.rs/ragit-korean
README
ragit-korean
Ragit-korean is a very simple korean tokenizer.
Ragit used to use charabia to tokenize cjk documents, but it has too many issues.
- Charabia bundles cjk dictionaries in the binary, which makes the file 70MiB bigger.
- It silently converts 완성형 korean to 조합형 korean. That silently messes up tfidf searches.