parquet-format-async-temp
| Crates.io | parquet-format-async-temp |
| lib.rs | parquet-format-async-temp |
| version | 0.3.1 |
| created_at | 2021-08-08 15:43:26.959715+00 |
| updated_at | 2022-06-14 06:22:46.852487+00 |
| description | Temporary crate containing thrift library + parquet definitions compiled to support read+write async. |
| homepage | https://github.com/jorgecarleitao/parquet-format-rs |
| repository | https://github.com/jorgecarleitao/parquet-format-rs |
| max_upload_size | |
| id | 433185 |
| size | 476,703 |
Jorge Leitao (jorgecarleitao)
documentation
README
parquet-format-async-temp
This is a temporary crate containing a subset of rust's thirft library and parquet to support native async parquet read and write.
Specifically, it:
- supports
asyncread API (viafutures) - supports
asyncwrite API (viafutures) - the write API returns the number of written bytes
It must be used with the fork of thrift's compiler available at https://github.com/jorgecarleitao/thrift/tree/write_size .
Why
To read and write files with thrift (e.g. parquet) without commiting to a
particular runtime (e.g. tokio, hyper, etc.), the protocol needs to support
AsyncRead + AsyncSeek and AsyncWrite respectively.
To not require Seek and AsyncSeek on write, the protocol must
return the number of written bytes on its write_* API.
This crate addresses these two concerns for parquet. It is essentially:
- https://github.com/apache/thrift/pull/2426 applied on latest thrift
- a modification for the written bytes to be outputed on all
write_*APIs