pipe_downloader
| Crates.io | pipe_downloader |
| lib.rs | pipe_downloader |
| version | 0.8.2 |
| source | src |
| created_at | 2022-11-28 16:55:17.2581 |
| updated_at | 2023-05-08 16:25:20.044354 |
| description | Fast downloader + extractor for *.tar.gz, *tar.lz4 and *.tar.bz2 files. |
| homepage | |
| repository | |
| max_upload_size | |
| id | 724607 |
| size | 912,776 |
(scx1332)
documentation
README
pipe_downloader
This program is used to download, uncompress and untar files at once. So basically it does the same as command:
curl http://arvhive.tar.lz4 | lz4 -dc | tar -xv -C output_dir
result is the same as using (pipe_downloader will create directory if not empty):
pipe_downloader --url http://arvhive.tar.lz4 --output-dir output_dir
but it uses multiple cores better than piped commands like above. Additionally, it provides progress, that can be used by server to provide API. Implementation is nonblocking, based on threads (no async IO used)
It spawns 3 or more threads per download, one or more for downloading, one for decompressing and one for untaring (writing files also). It is using chunks, when chunks are filled with data they are gathered and when next chunk is full they are processed by unpacker.
No unsafe code used.
Currently, it supports:
- archive.tar.lz4
- archive.tar.gz
- archive.tar.bz2
- archive.tar.zst
- archive.tar.xz
- single_file.lz4
- single_file.gz
- single_file.bz2
- single_file.zst
- single_file.xz
As base mode of operation it used PARTIAL_CONTENT (206) http status code, which allows to continue operation when connection is lost (downloader waits patiently until chunk is available again). Downloader also supports normal GET (200) download when server does not support PARTIAL_CONTENT, but any network timeout/disconnection will lead to unrecoverable error then.
There is no option of restarting download after unrecoverable error or killed process.
- Cross compilation
cross build --release --target aarch64-unknown-linux-musl
- Test runs
cargo run --release --example pipe_downloader -- --output-dir linux-6.0.4 --url=https://mirrors.edge.kernel.org/pub/linux/kernel/v6.x/linux-6.0.4.tar.gz