pipe_downloader

This program is used to download, uncompress and untar files at once. So basically it does the same as command:

curl http://arvhive.tar.lz4 | lz4 -dc | tar -xv -C output_dir

result is the same as using (pipe_downloader will create directory if not empty):

pipe_downloader --url http://arvhive.tar.lz4 --output-dir output_dir

but it uses multiple cores better than piped commands like above. Additionally, it provides progress, that can be used by server to provide API. Implementation is nonblocking, based on threads (no async IO used)

It spawns 3 or more threads per download, one or more for downloading, one for decompressing and one for untaring (writing files also). It is using chunks, when chunks are filled with data they are gathered and when next chunk is full they are processed by unpacker.

No unsafe code used.

Currently, it supports:

• archive.tar.lz4
• archive.tar.gz
• archive.tar.bz2
• single_file.lz4
• single_file.gz
• single_file.bz2

As base mode of operation it used PARTIAL_CONTENT (206) http status code, which allows to continue operation when connection is lost (downloader waits patiently until chunk is available again). Downloader also supports normal GET (200) download when server does not support PARTIAL_CONTENT, but any network timeout/disconnection will lead to unrecoverable error then.

There is no option of restarting download after unrecoverable error or killed process.

1. Cross compilation

cross build --release --target aarch64-unknown-linux-musl

2. Test runs

cargo run --release --example pipe_downloader -- --output-dir linux-6.0.4 --url=https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.0.4.tar.gz