remozipsy - Remote Zip Sync

Overview

remozipsy is a crate enabling incremental synchronization of zip archives from a remote location. It avoids redownloading the entire archive, instead it only fetches files that have changed since the previous download (by comparing file hashes). This optimizes download times over slow connections.

This crate is designed to efficiently update large datasets consisting of many individual files, with periodic updates. The logic was originally written for Airshipper, the game launcher for Veloren, to update the game more efficiently, but has been split out for general use.

Features

• 🚀 Parallel file downloads from remote server to reduce latency
• 🗜️ Local CRC comparison to avoid redundant data transfers
• 🧰 Abstractable interfaces (RemoteZip, FileSystem) enabling custom storage backends
• 🛠️ Prebuilt implementations for reqwest and tokio::fs
• 📡 Designed for performance in low-bandwidth networks

Installation

Add the following dependency to your Cargo.toml:

[dependencies]
remozipsy = "0.2.0"

Usage

Example: Sync remote zip with local filesystem

See examples/sync_remote_zip.rs for a detailed example.

use remozipsy::{Config, Statemachine, reqwest::ReqwestRemoteZip, tokio::TokioLocalStorage};

#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let remote = ReqwestRemoteZip::with_url("https://getsamplefiles.com/download/zip/sample-1.zip")?;
    let local = TokioLocalStorage::new("./extract", Vec::new());
    let state = Statemachine::new(remote, local, Config::default());

    while let Some((progress, next_state)) = state.progress().await {
        state = next_state;
        println!("Progress: {progress:?}");
        tokio::task::yield_now().await;
    }
    Ok(())
}

The main idea is to progress the internal statemachine with statemachine.progress().

Algorithm Overview

1. Fetch zip’s central directory (zip metadata)
2. Traverse filesystem, compare existing files by CRC hash
3. Determine which files to download/delete
4. Download updated/new files in parallel (using tokio tasks)
5. Unzip downloaded files locally
6. Delete files no longer present

Why remozipsy works

1. Zip structure: The central directory located at the end of the file contains all metadata, seperated from the data itself, which makes it fairly easily to selectively fetch.
2. File hashes: The metadata includes CRC32 hashes of the uncompressed files, making it possible to tell if files are identical or not without downloading them beforehand.
3. Individual file compression: All files in a Zip archive are compressed individually, and are therefor able to be extracted individually.
4. HTTP Range requests: HTTP allows the RANGE header to only fetch part of a remote file, luckily many webservers (and github releases) support this feature.

For more details, see the documentation and explore examples.

Contributing

Pull Requests welcome! Start by checking open issues or feature requests in our GitLab repo.

License

MIT License or APACHE-2.0 License