# libdeflater > Rust bindings to [libdeflate](https://github.com/ebiggers/libdeflate), a high-performance > library for working with gzip/zlib/deflate data. [![Build Status](https://travis-ci.org/adamkewley/libdeflater.svg?branch=master)](https://travis-ci.org/adamkewley/libdeflater) [![Crates.io](https://img.shields.io/crates/v/libdeflater.svg?maxAge=2592000)](https://crates.io/crates/libdeflater) [![Documentation](https://docs.rs/libdeflater/badge.svg)](https://docs.rs/libdeflater) ``` libdeflater = "1.22.0" ``` `libdeflater` is a thin wrapper library around [libdeflate](https://github.com/ebiggers/libdeflate). Libdeflate is optimal in applications that have the input data up-front, or when (large) input datasets can be split into smaller chunks (e.g. genomic [bam](https://samtools.github.io/hts-specs/SAMv1.pdf) files, some object stores, specialized backends, game netcode packets). This is a thin library around libdeflate that: - Builds [libdeflate](https://github.com/ebiggers/libdeflate) from source (see [libdeflate-sys](libdeflate-sys)'s [build.rs](libdeflate-sys/build.rs) file) - Binds to libdeflate's C API with rust bindings (see [lib.rs](src/lib.rs)) - Contains high-level integration tests to ensure the bindings work (see [integration_test.rs](https://github.com/adamkewley/libdeflater/blob/master/tests/integration_test.rs)) - Contains [usage examples](https://github.com/adamkewley/libdeflater/tree/master/examples) and a [benchmark suite](https://github.com/adamkewley/libdeflater/tree/master/benches). The benchmark suite indicates a 2-3x speedup accross the Calgary and Canterbury corpuses (see [Benchmarks](#benchmarks) section) ⚠️ **Warning**: [libdeflate](https://github.com/ebiggers/libdeflate) is best-suited for *specialized* use-cases where you know the rough size range of your input up-front. You should use something like [flate2](https://github.com/alexcrichton/flate2-rs) if you want a general-purpose deflate library that supports streaming. # Examples Example source [here](examples). To run the examples: ```bash cargo run --example gz_compress.rs cargo run --example gz_decompress.rs ``` # Benchmarks Benchmark data is from both the [Calgary Corpus](https://en.wikipedia.org/wiki/Calgary_corpus), and the [Canterbury Corpus](http://corpus.canterbury.ac.nz/resources/cantrbry.zip). The benchmark tables below were made with this set of steps: ```bash wget http://www.data-compression.info/files/corpora/largecalgarycorpus.zip unzip -d bench_data largecalgarycorpus.zip wget http://corpus.canterbury.ac.nz/resources/cantrbry.zip unzip -d bench_data cantrbry.zip # runs benchmarks against all files in `bench_data` cargo bench scripts/process-bench.rb encode scripts/process-bench.rb decode ``` ### Compression Avg. speedup (on this corpus) is around 2-3x ``` bench size [KB] speedup flate2 [us] libdeflate [us] alice29.txt 152 3.1 5636 1821 asyoulik.txt 125 3.1 4911 1584 bib 111 2.9 3278 1133 book1 768 3.2 32697 10377 book2 610 2.8 19780 6975 cp.html 24 2.3 394 170 fields.c 11 2.4 155 65 geo 102 7.1 7717 1082 grammar.lsp 3 2.0 38 19 kennedy.xls 1029 7.3 46598 6427 lcet10.txt 426 3.0 14924 4931 news 377 2.5 10160 4052 obj1 21 2.6 385 149 obj2 246 3.5 7771 2218 paper1 53 2.4 1312 543 paper2 82 2.7 2608 955 paper3 46 2.5 1303 513 paper4 13 2.2 226 102 paper5 11 2.1 182 88 paper6 38 2.3 848 367 pic 513 3.8 7508 1990 plrabn12.txt 481 3.4 22527 6698 progc 39 2.4 882 361 progl 71 2.8 1553 559 progp 49 2.6 904 346 ptt5 513 3.8 7389 1964 sum 38 3.8 1124 297 trans 93 2.5 1595 650 xargs.1 4 1.9 40 21 ``` ### Decompression Avg. speedup (on this corpus) is around 2x. ``` bench size [KB] speedup flate2 [us] libdeflate [us] alice29.txt 152 3.0 338 114 asyoulik.txt 125 2.8 300 106 bib 111 3.2 240 76 book1 768 2.5 1906 768 book2 610 2.7 1376 501 cp.html 24 2.0 31 16 fields.c 11 2.1 15 7 geo 102 2.3 359 160 grammar.lsp 3 1.8 7 4 kennedy.xls 1029 1.4 1241 911 lcet10.txt 426 2.8 919 325 news 377 2.4 969 400 obj1 21 1.9 41 21 obj2 246 2.5 558 220 paper1 53 3.2 109 34 paper2 82 3.1 182 58 paper3 46 3.0 100 34 paper4 13 1.9 22 12 paper5 11 1.9 21 11 paper6 38 3.0 75 25 pic 513 3.1 617 198 plrabn12.txt 481 2.5 1183 472 progc 39 3.1 76 25 progl 71 3.5 103 30 progp 49 3.0 65 22 ptt5 513 3.1 616 197 sum 38 2.5 75 31 trans 93 3.7 131 36 xargs.1 4 1.8 9 5 ``` ### Benchmark Notes - All benchmarks are single-threaded - IO/streaming overhead is not considered. The decompressed data is read into memory before performing the comparison - Comparison made against `flate2` with no feature flags (i.e. `miniz` implementation). `flate2` was chosen because it's the most popular. - Comparisons with other `flate2` backends are available on the `bench-flate2-miniz-oxide` and `bench-flate2-zlib` branches. The `zlib` backend is ~8 % faster on some of the corpus entries. - Compression performed with default compression setting in both cases - Corpus entries were compressed with `flate2` at default compression level ### Compile-time features You can enable the following features to customise the build: - `use_rust_alloc`: Makes libdeflate use Rust's allocator instead of the libc one. This is useful when Rust is preconfigured to use a [custom global allocator](https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/global-allocators.html) (e.g. pool-based, or a tracking one, or something else entirely). - `freestanding`: Builds libdeflate in a freestanding mode (no reliance on libc). This is useful for targets that don't have a C stdlib (e.g. `wasm32-unknown-unknown`) as otherwise they would fail to compile. Implies `use_rust_alloc`.