Crates.io | vortex |
lib.rs | vortex |
version | 0.20.0 |
source | src |
created_at | 2020-06-09 09:53:43.612656 |
updated_at | 2024-11-26 19:51:20.329263 |
description | Vortex file format with all builtin codecs and a sampling compressor. |
homepage | https://github.com/spiraldb/vortex |
repository | https://github.com/spiraldb/vortex |
max_upload_size | |
id | 251841 |
size | 19,152 |
Vortex is an extensible, state-of-the-art columnar file format, with associated tools for working with compressed Apache Arrow arrays in-memory, on-disk, and over-the-wire.
Vortex is an aspiring successor to Apache Parquet, with dramatically faster random access reads (100-200x faster) and scans (2-10x faster), while preserving approximately the same compression ratio and write throughput as Parquet with zstd. It is designed to support very wide tables (at least 10s of thousands of columns) and (eventually) on-device decompression on GPUs.
Vortex is intended to be to columnar file formats what Apache DataFusion is to query engines: highly extensible, extremely fast, & batteries-included.
[!CAUTION] This library is still under rapid development and is a work in progress!
Some key features are not yet implemented, both the API and the serialized format are likely to change in breaking ways, and we cannot yet guarantee correctness in all cases.
The major features of Vortex are:
One of the core design principles in Vortex is strict separation of logical and physical concerns.
For example, a Vortex array is defined by a logical data type (i.e., the type of scalar elements) as well as a physical encoding (the type of the array itself). Vortex ships with several built-in encodings, as well as several extension encodings.
The built-in encodings are primarily designed to model the Apache Arrow in-memory format, enabling us to construct
Vortex arrays with zero-copy from Arrow arrays. There are also several built-in encodings (e.g., sparse
and
chunked
) that are useful building blocks for other encodings. The included extension encodings are mostly designed
to model compressed in-memory arrays, such as run-length or dictionary encoding.
Analogously, vortex-serde
is designed to handle the low-level physical details of reading and writing Vortex arrays. Choices
about which encodings to use or how to logically chunk data are left up to the Compressor
implementation.
One of the unique attributes of the (in-progress) Vortex file format is that it encodes the physical layout of the data within the file's footer. This allows the file format to be effectively self-describing and to evolve without breaking changes to the file format specification.
For example, the Compressor implementation can choose to chunk data into a Parquet-like layout with row groups and aligned pages (ChunkedArray of StructArray of ChunkedArrays with equal chunk sizes). Alternatively, it can choose to chunk different columns differently based on their compressed size and data distributions (e.g., a column that is constant across all rows can be a single chunk, whereas a large string column may be split arbitrarily many times).
In the same vein, the format is designed to support forward compatibility by optionally embedding WASM decoders directly into the files themselves. This should help avoid the rapid calcification that has plagued other columnar file formats.
The Vortex type-system is still in flux. The current set of logical types is:
Vortex includes a base set of "flat" encodings that are designed to be zero-copy with Apache Arrow. These are the canonical representations of each of the logical data types. The canonical encodings currently supported are:
Vortex includes a set of highly data-parallel, vectorized encodings. These encodings each correspond to a compressed in-memory array implementation, allowing us to defer decompression. Currently, these are:
Vortex's default compression strategy is based on the BtrBlocks paper.
Roughly, for each chunk of data, a sample of at least ~1% of the data is taken. Compression is then attempted (recursively) with a set of lightweight encodings. The best-performing combination of encodings is then chosen to encode the entire chunk. This sounds like it would be very expensive, but given the logical types and basic statistics about a chunk, it is possible to cheaply prune many encodings and ensure the search space does not explode in size.
Vortex provides the ability for each encoding to specialize the implementation of a compute function to avoid decompressing where possible. For example, filtering a dictionary-encoded UTF8 array can be more cheaply performed by filtering the dictionary first.
Note--as mentioned above--that Vortex does not intend to become a full-fledged compute engine, but rather to implement basic compute operations as may be required for efficient scanning & pushdown.
Vortex arrays carry lazily-computed summary statistics. Unlike other array libraries, these statistics can be populated from disk formats such as Parquet and preserved all the way into a compute engine. Statistics are available to compute kernels as well as to the compressor.
The current statistics are:
The goals of the vortex-serde
implementation are:
TODO: insert diagram here
Apache Arrow is the de facto standard for interoperating on columnar array data. Naturally, Vortex is designed to be maximally compatible with Apache Arrow. All Arrow arrays can be converted into Vortex arrays with zero-copy, and a Vortex array constructed from an Arrow array can be converted back to Arrow, again with zero-copy.
It is important to note that Vortex and Arrow have different--albeit complementary--goals.
Vortex explicitly separates logical types from physical encodings, distinguishing it from Arrow. This allows
Vortex to model more complex arrays while still exposing a logical interface. For example, Vortex can model a UTF8
ChunkedArray
where the first chunk is run-length encoded and the second chunk is dictionary encoded.
In Arrow, RunLengthArray
and DictionaryArray
are separate incompatible types, and so cannot be combined in this way.
For best performance we recommend using MiMalloc as the application's allocator.
#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;
Please see CONTRIBUTING.md.
The project has several optional-but-recommended external dependencies:
# Required if you want to modify any of the .fbs or .proto files
brew install flatbuffers protobuf
# Required for benchmarks
brew install duckdb
You also need the Rust toolchain installed. If you haven't already, install rustup with one of the following commands:
# option 1
brew install rustup
# option 2
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
This repo uses uv to manage the combined Rust/Python monorepo build. After installing uv, make sure to run:
# Install uv from https://docs.astral.sh/uv/getting-started/installation/
uv sync
Licensed under the Apache License, Version 2.0 (the "License").
Vortex is and will remain an open-source project. Our intent is to model its governance structure after the Substrait project, which in turn is based on the model of the Apache Software Foundation. Expect more details on this in Q4 2024.
This project is inspired by and--in some cases--directly based upon the existing, excellent work of many researchers and OSS developers.
In particular, the following academic papers have strongly influenced development:
Additionally, we benefited greatly from:
Thanks to all of the aforementioned for sharing their work and knowledge with the world! 🚀