znippy
| Crates.io | znippy |
| lib.rs | znippy |
| version | 0.2.4 |
| created_at | 2025-07-25 10:18:32.682706+00 |
| updated_at | 2025-07-30 09:52:43.878093+00 |
| description | Znippy compresses files using Arrow IPC. Readable by DuckDB,Polars,Datafusion. May Allfather Odin watch over every bit. |
| homepage | |
| repository | https://github.com/Ignalina/znippy |
| max_upload_size | |
| id | 1767391 |
| size | 49,992 |
Rickard Ernst Björn Lundin (Ignalina)
documentation
README
Znippy
Znippy archive format based on Zstandard and Apache Arrow . Built for speed, streaming, and random access, it was originally created for Artisan—the airgap-driven artifact toolbox—and now stands as a fast, interchangeable backend.
| ⚡ Znippy | 🐢 tar + zstd | |
|---|---|---|
| ⚡ Compression | 44.6 seconds 2767 MB → 911 MB 418 % compression ratio |
63.3 seconds 2767 MB → 722 MB |
| ⚡ Decompression | 1.9 seconds 14979 chunks fully indexed |
3.9 seconds streamed extraction |
Snippy is dead — long live Znippy!
The name Snippy was already in use in the genomics world, so we’re moving forward with Znippy — a name that proudly reflects its foundation in Zstandard (Zstd) and its focus on real-time, high-performance compression.
🧩 Znippy Compression Pipeline – Visual Overview
flowchart LR
%% Reader + ChunkRevolver
Reader[Reader Thread] --> Revolver((ChunkRevolver))
%% Fan-out
Revolver --> C0[Compressor 0]
Revolver --> C1[Compressor 1]
Revolver --> C2[...]
Revolver --> C31[Compressor 31]
%% Fan-in
C0 --> Writer[Writer Thread]
C1 --> Writer
C2 --> Writer
C31 --> Writer
%% Output and Index
Writer --> Index[Arrow Index - znippy]
Writer --> Zdata[zdata File]
%% Microchunk entries in index
Index --> M0[Microchunk 0]
Index --> M1[Microchunk 1]
Index --> M2[...]
Index --> M63[Microchunk 63]
M0 --> Zdata
M1 --> Zdata
M2 --> Zdata
M63 --> Zdata
%% Final archive
Index --> Final[Znippy Archive Output]
Zdata --> Final