Parallel Downloader (pd) 🦀

A robust, concurrent file downloader built in Rust. It is designed to be resilient, supporting automatic retries, crash recovery, and download verification.

📋 Table of Contents

• 🚀 Features
• 🔧 How It Works
• 📦 Installation
  • Pre-built Binaries
  • Build from Source
• 🛠 Usage
• Options
• Commands
• Configuration
• 📚 Library Usage
• 🧪 Testing
• 📂 Examples
  • Simple Download
• 🔧 Troubleshooting
• 🤝 Contributing
• 📄 License

🚀 Features

• Concurrency: Downloads files in parallel chunks to maximize bandwidth.
• Resiliency: Automatically retries failed chunks on network timeouts.
• Crash Recovery: Saves progress to a state file (.state.json). If the program crashes or is interrupted, simply run the command again to resume exactly where it left off.
• Rate Limiting: Optional token-bucket throttling to limit bandwidth usage.
• Integrity Check: Verifies the final file against a SHA-256 hash.
• Library Support: Logic is separated from the CLI, allowing you to use the core downloader in your own Rust applications.
• Batch Processing: Supports input files (-i list.txt) for bulk downloads.
• Background Daemon: Run downloads in the background with job management.
• Interactive TUI: Terminal-based dashboard for monitoring downloads.
• Job Control: Pause and resume individual downloads in daemon mode.
• Authentication: Optional server secret for securing daemon access.

🔧 How It Works

Parallel Downloader uses a chunk-based parallel downloading approach:

1. File Analysis: First, it determines the total file size using HTTP HEAD requests
2. Chunk Division: The file is divided into equal-sized chunks (typically based on thread count)
3. Parallel Download: Each chunk is downloaded concurrently using HTTP Range requests
4. Progress Tracking: Individual progress bars show download status for each chunk
5. State Persistence: Download progress is saved to *.state.json files for resumability
6. Integrity Verification: Optional SHA-256 verification ensures file correctness

This approach maximizes bandwidth utilization while providing robust error recovery and resumability.

📦 Installation

Pre-built Binaries

Download the latest release for Windows, Linux, or macOS from the Releases Page.

Build from Source

git clone https://github.com/velumuruganr/parallel_downloader.git
cd parallel_downloader
cargo install --path .

Note: The CLI binary is provided in src/bin/pd/main.rs (the CLI glue is in src/bin/pd/). When developing from source you can run the binary directly with:

# Run the library's CLI binary from source
cargo run --bin pd -- run --url "https://example.com/large.iso"

🛠 Usage

1. Standalone Mode (CLI)

Best for quick, one-off downloads.

# Simple download (defaults to 4 threads)
pd run --url "https://example.com/large.iso"

# Advanced Usage
pd run \
  --url "https://example.com/movie.mp4" \
  --output "holiday.mp4" \
  --threads 8 \
  --rate-limit 1048576 \
  --dir ./downloads

2. Batch Mode

Download a list of URLs (one per line).

# Download files from list.txt, 3 files at a time
pd run -i list.txt -c 3

3. Daemon Mode (Background Service)

Best for long-running servers or managing queues.

# 1. Start the daemon in a separate terminal
pd start

# 2. Add downloads from any terminal
pd add --url "https://example.com/file1.zip"
pd add --url "https://example.com/file2.zip"

# 3. Check status
pd status

# 4. Control downloads
pd pause 1    # Pause download with ID 1
pd resume 1   # Resume download with ID 1

# 5. Interactive dashboard
pd tui        # Open terminal user interface

# 6. Shutdown
pd stop

Options

Commands

Configuration

This project supports reading settings from a config.toml file or from environment variables prefixed with PD.

• File locations (platform-specific):

  • Linux: ~/.config/pd/config.toml
  • macOS: ~/Library/Application Support/pd/config.toml
  • Windows: %APPDATA%\pd\config.toml

• Example: copy config.example.toml to one of the locations above and edit values such as threads, rate_limit, default_dir, concurrent_files, server_secret, and server_addr.

• Environment variables: you can override values via environment variables using __ as a separator. For example:

# set threads to 8
export PD__threads=8

# set global rate limit to 1 MiB/s
export PD__rate_limit=1048576

# set server secret for daemon authentication
export PD__server_secret="my-secret-key"

# set server bind address
export PD__server_addr="0.0.0.0"

Environment variables take precedence over values found in config.toml.

📚 Library Usage

You can use parallel_downloader as a library in your own project.

Add to your Cargo.toml:

[dependencies]
parallel_downloader = { version = "0.3.0", default-features = false }

Use the modules in your code:

use parallel_downloader::downloader::prepare_download;
use parallel_downloader::worker::download_chunk;

🧪 Testing

We use wiremock to simulate HTTP failures and chunk ranges without hitting real servers.

cargo test

📂 Examples

The repository includes runnable examples in the examples/ folder to demonstrate how to use parallel_downloader programmatically in your own applications.

Simple Download

This example demonstrates a complete workflow: creating a client, verifying file size, and downloading chunks with a progress bar.

To run the example:

cargo run --example simple_download

You can view the full source code in examples/simple_download.rs.

🔧 Troubleshooting

Common Issues

Download fails with "Connection reset" errors

• Try reducing the number of threads: pd run --threads 2 --url <URL>
• Check your internet connection stability
• Some servers may not support concurrent connections

State file corruption

• Delete the .state.json file and restart the download
• The state file is created alongside your download with the same name

Daemon connection refused

• Ensure the daemon is running: pd start
• Check if the port is available (default: 127.0.0.1:9090)
• Verify server address in config if changed

High CPU usage

• Reduce thread count or add rate limiting
• This is normal for parallel downloads

Permission denied errors

• Check write permissions in the target directory
• Use sudo if downloading to system directories

Getting Help

• Check existing GitHub Issues
• Create a new issue with:
  • Your OS and Rust version
  • Command used and full error output
  • URL (anonymized if needed)

🤝 Contributing

We welcome contributions! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

Development Setup

1. Fork the repository
2. Clone your fork: git clone https://github.com/velumuruganr/parallel_downloader.git
3. Create a feature branch: git checkout -b feature/amazing-feature
4. Make your changes and add tests
5. Run tests: cargo test
6. Commit your changes: git commit -m 'Add amazing feature'
7. Push to the branch: git push origin feature/amazing-feature
8. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.