newline_normalizer
| Crates.io | newline_normalizer |
| lib.rs | newline_normalizer |
| version | 0.1.6 |
| created_at | 2025-04-25 18:35:23.62159+00 |
| updated_at | 2025-04-25 22:07:02.329623+00 |
| description | Zero-copy newline normalization to \n or \r\n with SIMD acceleration. |
| homepage | |
| repository | https://github.com/digitalcortex/newline_normalizer |
| max_upload_size | |
| id | 1649347 |
| size | 36,755 |
Nikita Madebeykin (digitalcortex)
documentation
README
🧹 newline_normalizer 🧹
The library for normalizing text into Unix (\n) or DOS (\r\n) newline formats, using fast SIMD search and zero-copy when possible.
✨ Features
- Adds extension traits to
str— call.to_unix_newlines()and.to_dos_newlines()directly. - Preserves input with
Cow<str>— skips allocation if no changes are needed. - Converts
\rand\r\ninto consistent Unix (\n) or DOS (\r\n) newlines. - Unicode-safe — preserves all characters without loss.
- Fast scanning with memchr and SIMD.
📚 Examples
use newline_normalizer::{ToUnixNewlines, ToDosNewlines};
let unix = "line1\r\nline2\rline3".to_unix_newlines();
assert_eq!(unix, "line1\nline2\nline3");
let dos = "line1\nline2\nline3".to_dos_newlines();
assert_eq!(dos, "line1\r\nline2\r\nline3");
🚀 Benchmark
Benchmarks are in the /benches folder.
Run them using:
cargo bench --bench to_unix
cargo bench --bench to_dos
All suggestions on how to improve the benchmarks are welcome.
📈 Results
Hardware: AMD Ryzen 9 9900X 12-Core Processor with 64 GB RAM.
Rust version: rustc 1.86.0 (05f9846f8 2025-03-31)
Benchmark framework: Criterion
Normalizing to DOS newlines (\r\n):
| Case | newline-converter |
This crate (newline_normalizer) |
|---|---|---|
| Small Unicode paragraph | ~685.46 ns | ~88.789 ns 🚀 |
| Small Unicode paragraph pre-normalized | ~151.39 ns | ~58.350 ns 🚀 |
| The Adventures of Sherlock Holmes (608kb) | ~345.27 µs | ~138.26 µs 🚀 |
| The Adventures of Sherlock Holmes (608kb) pre-normalized | ~342.91 µs | ~137.54 µs 🚀 |
Note: Pre-normalized means the input already has correct line endings and does not require changes.
Normalizing to Unix newlines (\n):
| Case | newline-converter |
regex replace all |
This crate (newline_normalizer) |
|---|---|---|---|
| Small Unicode paragraph | ~1.0858 µs | ~101.42 ns | ~24.464 ns 🚀 |
| Small Unicode paragraph pre-normalized | ~164.41 ns | ~20.744 ns | ~4.6608 ns 🚀 |
| The Adventures of Sherlock Holmes (608kb) | ~680.12 µs | ~289.84 µs | ~89.150 µs 🚀 |
| The Adventures of Sherlock Holmes (608kb) pre-normalized | ~318.83 µs | ~7.7864 µs | ~2.5146 µs 🚀 |
Benchmark result notes
- Pre-normalized means the input text already uses the correct line endings.
- In such cases,
newline_normalizercan skip allocations and return a borrowed reference. - Extremely low latency (e.g., ~4.66 ns) is achieved by using
Cow::Borrowed, avoiding an allocation of a new string when the input does not change.
🔤 Unicode behavior
This crate does not alter Unicode content. It only rewrites newline boundaries.
All valid UTF-8 sequences are preserved, including:
- Combining characters stay attached
- Emoji and multi-codepoint sequences remain valid
- Right-to-left (RTL) markers are unaffected
⚠️ Limitations
This crate does not currently normalize U+2028 (LINE SEPARATOR) or U+2029 (PARA SEP). Only ASCII newline formats are converted.
📝 Licensed under MIT
This project is licensed under the MIT License.