Cargo-Crate-Downloader

This project allows you to download source code of a single Rust crate from crates.io or download the entire crates.io to local. We provide both library and runnable codes to do so.

Download the entire crates.io

For test and direct usage, you can use another binary that will download a single crate into the target directory:

// For example: cargo run --bin download_one -- rusqlite 0.29.0
cargo run --bin download_one -- <name> <version_num>

We provide a demo for you to learn how to download crates.

use std::fs::{create_dir, remove_dir_all};
use std::path::Path;

use downloader::Downloader;

use crate_downloader::{deal_with_crate, fetch_crate};

const CRATESDIR: &str = "./demo";


fn main() {
    let name = "rand";
    let version_num = "0.8.5";

    // Main Process
    println!("Processing crate {name}-v{version_num}");
    remove_dir_all(CRATESDIR).unwrap_or_default(); // Delete tmp crates file directory
    create_dir(Path::new(CRATESDIR)).unwrap_or_default(); // Crates file directory
    let mut downloader = Downloader::builder()
        .download_folder(Path::new(CRATESDIR))
        .parallel_requests(1)
        .build()
        .expect("Fatal Error, build downloader fails!");

    if let Err(e) =
        fetch_crate( &mut downloader, CRATESDIR, &name, &version_num)
    {
        println!("Fetch fails: {}",  e);
    } else if let Err(e) = deal_with_crate(CRATESDIR, &name, &version_num) {
        println!("Unzip fails: {}", e);
    } else {
        println!("Success. Check the directory ./{}", CRATESDIR);
    }

}

Download the entire crates.io

The crate version info are based on DB of our Cargo-Ecosystem-Monitor. Follow the guide in the root directory.

In this project, we will download and decompress all crates source code from crates.io (Cargo Ecosystem). The process can be broken and re-started, as we support break-point execution. The data will be stored in CRATESDIR(./on_process by default, see code to change it).

Directly running the project requires preliminary DB support. Make sure you are prepared to do so.

cargo run