lzfse_rust

Rust LZFSE implementation.

Documentation

https://docs.rs/lzfse_rust

Install

Simply configure your Cargo.toml:

[dependencies]
lzfse_rust = "0.2"

Overview

This crate provides two LZFSE engines: one operating over user supplied memory buffers, and one operating over internal ring buffers.

The memory buffered engine works directly with input and output buffers that we supply. It is exposed via LzfseEncoder and LzfseDecoder objects. We would consider this engine when operating on &[u8] and Vec<u8> objects.

The ring buffered engine works by streaming data in and out of it's internal ring buffers. It is exposed via LzfseRingEncoder and LzfseRingDecoder objects. We would consider this engine when operating on IO streams, or when we want to expose a Read or Write interface.

Check the documentation for additional information and examples.

Examples

This program compresses data from stdin into stdout. This example can be found in examples/compress_ring.rs

use lzfse_rust::LzfseRingEncoder;
use std::io;

fn main() -> io::Result<()> {
    let mut rdr = io::stdin();
    let mut wtr = io::stdout();
    let mut encoder = LzfseRingEncoder::default();
    encoder.encode(&mut rdr, &mut wtr)?;
    Ok(())
}

This program decompresses data from stdin into stdout. This example can be found in examples/decompress_ring.rs

use lzfse_rust::LzfseRingDecoder;
use std::io;

fn main() -> io::Result<()> {
    let mut rdr = io::stdin();
    let mut wtr = io::stdout();
    let mut decoder = LzfseRingDecoder::default();
    decoder.decode(&mut rdr, &mut wtr)?;
    Ok(())
}

Command line tool: lzfoo

A fast, memory efficient and stream capable lzfse command line tool clone. Powered by lzfse_rust.

Install.

$ cargo install lzfoo

Compress a.txt to a.txt.lzfse:

$ lzfoo -encode -i a.txt -o a.txt.lzfse

Compress with stdin/ stdout:

$ lzfoo -encode -i < a.txt > a.txt.lzfse

$ echo "semper fidelis" | lzfoo -encode > a.txt.lzfse

Decompress a.txt.lzfse to a.txt:

$ lzfoo -decode -i a.txt.lzfse -o a.txt

Decompress with stdin/ stdout:

$ lzfoo -decode -i < a.txt.lzfse > a.txt

Check the lzfoo crate for details.

Testing

This crate comes with a comprehensive test suite that is divided into unit tests and integration tests. The unit tests check that the library's internal components are working, whilst the integration tests check that library as a whole is working. Extended tests are available but may take hours to complete.

Unit tests:

$ cargo test

Unit tests, extended:

$ cargo test -- --ignored

Integration tests:

$ cargo test --manifest-path test/Cargo.toml

Integration tests, extended:

$ cargo test --manifest-path test/Cargo.toml -- --ignored

Additional integration tests are available. Notably, validation tests with the reference LZFSE implementation. These are described in test, along with instructions on how to build and run them.

Performance

As a library and with the stated test machine, lzfse_rust outperforms lzfse_ref. However results will vary with different machines.

The benchmarks are powered with Criterion and formatted with critcmp. The machine is an 8GB Intel i5-2500K running Ubuntu 18.04 (64 bit). The dataset is taken from Snappy. The benchmark source is here, along with instructions on how to build and run the benchmarks.

group                          new/lzfse_ref/                         new/rust/                              new/rust_ring/
-----                          --------------                         ---------                              --------------
decode/snap_uflat00_html       1.24    119.6±0.11µs   816.8 MB/sec    1.07    103.3±0.03µs   945.7 MB/sec    1.00     96.2±0.04µs  1014.9 MB/sec
decode/snap_uflat01_urls       1.18   1407.8±3.84µs   475.6 MB/sec    1.02   1211.8±0.11µs   552.5 MB/sec    1.00   1193.0±3.27µs   561.2 MB/sec
decode/snap_uflat02_jpg        1.07    353.7±0.05µs   331.9 MB/sec    1.00    330.7±0.91µs   355.0 MB/sec    1.02    336.1±0.85µs   349.3 MB/sec
decode/snap_uflat04_pdf        1.07    243.6±0.07µs   400.8 MB/sec    1.00    227.7±0.04µs   429.0 MB/sec    1.00    228.7±0.04µs   427.0 MB/sec
decode/snap_uflat05_html4      1.13    140.1±0.08µs     2.7 GB/sec    1.00    123.4±0.36µs     3.1 GB/sec    1.12    138.4±0.04µs     2.8 GB/sec
decode/snap_uflat06_txt1       1.22    469.4±0.05µs   309.0 MB/sec    1.09    420.8±1.21µs   344.7 MB/sec    1.00    384.3±0.05µs   377.4 MB/sec
decode/snap_uflat07_txt2       1.20    410.2±1.18µs   291.0 MB/sec    1.10    373.4±0.01µs   319.7 MB/sec    1.00    340.9±0.02µs   350.2 MB/sec
decode/snap_uflat08_txt3       1.25   1255.5±0.12µs   324.2 MB/sec    1.09   1096.9±3.32µs   371.0 MB/sec    1.00   1006.9±0.15µs   404.2 MB/sec
decode/snap_uflat09_txt4       1.18   1628.8±0.25µs   282.1 MB/sec    1.10   1511.5±3.03µs   304.0 MB/sec    1.00   1376.4±0.11µs   333.9 MB/sec
decode/snap_uflat10_pb         1.17    101.7±0.04µs  1112.3 MB/sec    1.04     90.2±0.03µs  1254.1 MB/sec    1.00     86.7±0.04µs  1304.3 MB/sec
decode/snap_uflat11_gaviota    1.28    486.0±0.05µs   361.7 MB/sec    1.09    413.2±0.05µs   425.4 MB/sec    1.00    379.5±0.03µs   463.1 MB/sec
encode/snap_uflat00_html       1.83   1500.0±3.76µs    65.1 MB/sec    1.00    821.1±0.09µs   118.9 MB/sec    1.10    905.9±0.11µs   107.8 MB/sec
encode/snap_uflat01_urls       1.45     13.1±0.00ms    51.3 MB/sec    1.00      9.0±0.00ms    74.2 MB/sec    1.01      9.1±0.00ms    73.8 MB/sec
encode/snap_uflat02_jpg        1.11      2.1±0.01ms    55.4 MB/sec    1.00   1910.0±5.33µs    61.5 MB/sec    1.12      2.1±0.00ms    54.7 MB/sec
encode/snap_uflat04_pdf        1.11   1695.1±0.19µs    57.6 MB/sec    1.00   1528.3±0.15µs    63.9 MB/sec    1.12   1705.8±0.16µs    57.3 MB/sec
encode/snap_uflat05_html4      5.10      4.4±0.00ms    89.7 MB/sec    1.00    854.4±0.08µs   457.2 MB/sec    1.12    954.2±0.11µs   409.4 MB/sec
encode/snap_uflat06_txt1       1.36      3.6±0.01ms    40.4 MB/sec    1.00      2.6±0.00ms    55.1 MB/sec    1.01      2.7±0.01ms    54.6 MB/sec
encode/snap_uflat07_txt2       1.34      3.1±0.01ms    38.5 MB/sec    1.01      2.3±0.00ms    51.2 MB/sec    1.00      2.3±0.00ms    51.8 MB/sec
encode/snap_uflat08_txt3       1.37      9.5±0.00ms    42.6 MB/sec    1.00      7.0±0.01ms    58.5 MB/sec    1.02      7.1±0.02ms    57.6 MB/sec
encode/snap_uflat09_txt4       1.34     12.3±0.04ms    37.3 MB/sec    1.00      9.3±0.00ms    49.7 MB/sec    1.00      9.2±0.00ms    49.9 MB/sec
encode/snap_uflat10_pb         1.95   1568.4±3.90µs    72.1 MB/sec    1.00    802.5±0.08µs   140.9 MB/sec    1.14    915.6±0.11µs   123.5 MB/sec
encode/snap_uflat11_gaviota    1.49      3.5±0.00ms    50.2 MB/sec    1.00      2.4±0.00ms    74.8 MB/sec    1.01      2.4±0.00ms    73.9 MB/sec

Key:
lzfse_ref: lzfse reference library
rust     : lzfse_rust
rust_ring: lzfse_rust ring

Column: 1 2 3
1: relative time                    lower is better, 1.00 is the fastest 
2: mean time ± standard deviation   lower is better
3: throughput                       higher is better

Minimum Rust version policy

This crate's minimum supported rustc version is 1.51.0.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Alternatives

• lzfse-rs - bindings to the reference LZFSE implementation.