Butterfly-dl 🦋

A high-performance, memory-efficient OpenStreetMap data downloader with intelligent source routing, resilient networking, and beautiful progress display.

Features

• 🚀 Optimized for Large Files: <1GB RAM usage regardless of file size (including 81GB planet.osm.pbf)
• 🎨 Enhanced Progress Display: Beautiful tqdm-style progress bars with smooth Unicode blocks
• 🛡️ Network Resilience: Intelligent retry with exponential backoff and smart resume from interruption points
• 📁 File Safety: Comprehensive overwrite protection with prompts and CLI flags
• 🧠 Smart Source Routing: HTTP with parallel downloads optimized by file size
• 🔍 Semantic Error Intelligence: Advanced fuzzy matching that understands semantic intent and geographic relationships
• 🌍 Dynamic Source Loading: Automatically fetches latest available regions from Geofabrik
• 📡 HTTP Protocol: Advanced HTTP with range requests and connection pooling
• 💧 Streaming Support: Direct stdout streaming for pipeline integration
• ⚡ Performance Optimized: Auto-tuning connections, Direct I/O for large files
• 🔧 Curl-like Interface: Simple positional arguments, stderr logging

Installation

Pre-built Binaries (Recommended)

Download the latest release for your platform:

# Linux (x86_64)
wget https://github.com/butterfly-osm/butterfly-osm/releases/latest/download/butterfly-dl-v2.0.0-x86_64-linux.tar.gz
tar -xzf butterfly-dl-v2.0.0-x86_64-linux.tar.gz
sudo mv butterfly-dl-v2.0.0-x86_64-linux/butterfly-dl /usr/local/bin/

# macOS (Intel)
wget https://github.com/butterfly-osm/butterfly-osm/releases/latest/download/butterfly-dl-v2.0.0-x86_64-macos.tar.gz
tar -xzf butterfly-dl-v2.0.0-x86_64-macos.tar.gz
sudo mv butterfly-dl-v2.0.0-x86_64-macos/butterfly-dl /usr/local/bin/

# Windows
# Download butterfly-dl-v2.0.0-x86_64-windows.zip from releases page

Package Managers

Debian/Ubuntu

wget https://github.com/butterfly-osm/butterfly-osm/releases/latest/download/butterfly-dl_2.0.0_amd64.deb
sudo dpkg -i butterfly-dl_2.0.0_amd64.deb

Cargo (Rust)

cargo install butterfly-dl

Build from Source

git clone https://github.com/butterfly-osm/butterfly-osm
cd butterfly-osm
cargo build --release -p butterfly-dl

Workspace Structure

This repository is organized as a Rust workspace containing multiple OSM tools:

• butterfly-common: Shared utilities and error handling
• tools/butterfly-dl: The main downloader tool
• Future tools: butterfly-shrink, butterfly-extract, butterfly-serve

Building Individual Tools

# Build all tools
cargo build --workspace --release

# Build specific tool
cargo build --release -p butterfly-dl

# Test specific tool
cargo test -p butterfly-dl

Usage

Basic Examples

# Download planet file from HTTP (81GB) 
butterfly-dl planet

# Download continent from HTTP
butterfly-dl europe

# Download country/region from HTTP  
butterfly-dl europe/belgium

# Stream to stdout for processing
butterfly-dl europe/monaco - | gzip > monaco.pbf.gz

# Save to custom file name
butterfly-dl planet planet-backup.pbf

# Verbose output with source info
butterfly-dl --verbose europe/belgium

Enhanced Features

🎨 Beautiful Progress Display

# Smooth tqdm-style progress bars with comprehensive information
butterfly-dl europe/belgium
# 75%|████████████▊     | 450MB/600MB [01:30<00:30, 25.2MB/s]

🛡️ Network Resilience & Recovery

# Automatic retry with smart resume - no lost progress
butterfly-dl europe/belgium
# ⚠️ Network error (attempt 1): operation timed out. Retrying in 1000ms...
# ⚠️ Stream interrupted at 300MB, resuming...
# ✅ Download completed!

📁 File Overwrite Protection

# Interactive prompts for existing files
butterfly-dl europe/belgium
# ⚠️ File already exists: belgium-latest.osm.pbf
# Overwrite? [y/N]: n
# ❌ Download cancelled

# Force overwrite without prompting
butterfly-dl europe/belgium --force
# ⚠️ Overwriting existing file: belgium-latest.osm.pbf

# Never overwrite, fail if file exists
butterfly-dl europe/belgium --no-clobber
# Error: File already exists: belgium-latest.osm.pbf (use --force to overwrite)

Source Resolution

Intelligent Error Handling

butterfly-dl includes smart error correction with fuzzy matching:

# Semantic intent recognition
butterfly-dl austrailia
# Error: Source 'austrailia' not found. Did you mean 'australia-oceania'?

# Typo correction
butterfly-dl antartica
# Error: Source 'antartica' not found. Did you mean 'antarctica'?

# Geographic accuracy  
butterfly-dl antartica/belgium
# Error: Source 'antartica/belgium' not found. Did you mean 'europe/belgium'?

# Standalone country recognition
butterfly-dl luxembourg
# Error: Source 'luxembourg' not found. Did you mean 'europe/luxembourg'?

# Smart continent suggestions
butterfly-dl plant
# Error: Source 'plant' not found. Did you mean 'planet'?

Features:

• Semantic Intelligence: Hybrid fuzzy matching that understands semantic intent, not just character distance
• Dynamic Source Discovery: Automatically fetches available regions from Geofabrik JSON API
• Contextual Scoring: Prioritizes meaningful matches like "australia-oceania" over "austria" for "austrailia"
• Geographic Intelligence: Knows Belgium belongs to Europe, not Antarctica
• Fallback Protection: Works offline with comprehensive fallback region list

Output Options

• No output argument: Auto-generated filename (e.g., belgium-latest.osm.pbf)
• Filename: Save to specified file
• -: Stream to stdout (logs go to stderr)

Performance Features

Memory Efficiency

• Fixed 64KB buffers: Memory usage independent of file size
• Ring buffer ordering: Small memory footprint for parallel downloads
• Direct I/O: Bypasses OS page cache for files >1GB (Unix systems)
• Streaming writes: No intermediate accumulation

Download Optimization

• HTTP: Single optimized stream for maximum network utilization
• HTTP: Auto-tuned parallel range requests (2-16 connections based on file size)
• Fallback: Graceful degradation for servers without range support
• Progress tracking: Real-time progress bars to stderr

Intelligent Defaults

• Connection scaling: Based on file size and CPU count
• Protocol selection: Optimal source for each data type
• Error handling: Robust retry and fallback mechanisms

Technical Architecture

Memory Usage Breakdown

Connection buffers:    16 × 64KB = 1MB
Ring buffer:          64MB (max)
HTTP client overhead: ~50MB
Runtime:             ~50MB
Total:               ~215MB (well under 1GB limit)

Direct I/O Support

Automatically enabled for files >1GB on Unix systems:

• Bypasses OS page cache
• Reduces memory pressure
• Optimizes large sequential writes
• Falls back gracefully if not available

CLI Reference

Downloads single OpenStreetMap files efficiently:
  butterfly-dl planet              # Download planet file (81GB) from HTTP
  butterfly-dl europe              # Download Europe continent from HTTP
  butterfly-dl europe/belgium      # Download Belgium from HTTP
  butterfly-dl europe/monaco -     # Stream Monaco to stdout

Usage: butterfly-dl [OPTIONS] <SOURCE> [OUTPUT]

Arguments:
  <SOURCE>  Source to download: "planet" (HTTP), "europe" (continent), or "europe/belgium" (country/region)
  [OUTPUT]  Output file path, or "-" for stdout

Options:
      --dry-run  Show what would be downloaded without downloading
  -v, --verbose  Enable verbose logging
  -h, --help     Print help
  -V, --version  Print version

Examples

Planet Download (81GB)

# Download planet file (uses HTTP, single stream, Direct I/O)
butterfly-dl planet

# Stream planet to compressed archive
butterfly-dl planet - | gzip > planet.pbf.gz

Regional Downloads

# Download all of Europe (parallel HTTP ranges)
butterfly-dl europe

# Download specific country
butterfly-dl europe/germany

# Download to custom location
butterfly-dl asia/japan japan-$(date +%Y%m%d).pbf

Pipeline Integration

# Stream and process immediately
butterfly-dl europe/monaco - | osmium fileinfo -

# Compress on the fly
butterfly-dl europe/switzerland - | bzip2 > switzerland.pbf.bz2

# Chain with other tools
butterfly-dl planet - | osmium extract --bbox 2.3,46.8,2.4,46.9 -o monaco-bbox.pbf -

Development

Building

cargo build --release

Testing

# Run all tests
cargo test

# Run with verbose output
cargo test -- --nocapture

Performance Testing

# Test with small file
time butterfly-dl europe/monaco

# Test streaming
butterfly-dl europe/monaco - | wc -c

Version Management

The project uses a centralized version management system to maintain consistency across all components:

📄 Single Source of Truth:

• VERSION file contains the current version number (e.g., 1.0.0)
• All other files automatically read from this central location

🔧 Automatic Version Propagation:

• CLI tool: Uses env!("BUTTERFLY_VERSION") from build script
• HTTP User-Agent: Dynamically includes version in requests
• Library exports: Version available via build-time environment
• C bindings: pkg-config file includes correct version
• Documentation: Version stays in sync automatically

🔄 Build Integration:

• build.rs reads VERSION file and sets environment variables
• Any change to VERSION triggers automatic rebuild
• Build system tracks version file as dependency

📝 Updating Version:

# Update version for new release
echo "1.1.0" > VERSION

# Rebuild automatically picks up new version
cargo build --release

# All components now use 1.1.0
./target/release/butterfly-dl --version  # Shows 1.1.0

Note: Cargo.toml version must still be updated manually due to Cargo limitations.

Architecture

• Rust + Tokio: Async/await for concurrent downloads
• HTTP Client: Advanced reqwest client with connection pooling
• Reqwest: HTTP client with connection pooling and range requests
• Indicatif: Progress bars to stderr
• Ring buffer: Maintains chunk ordering with minimal memory

Performance Benchmarks

butterfly-dl includes comprehensive benchmarking against industry-standard tools to validate performance claims:

Benchmark Suite

# Run benchmarks against curl and aria2
./benchmarks/bench.sh europe/monaco    # Small file (~1MB)
./benchmarks/bench.sh europe/belgium   # Medium file (~43MB)  
./benchmarks/bench.sh europe/france    # Large file (~3.5GB)

Sample Results

All benchmarks conducted on actual hardware over real network conditions

Small Files (Monaco ~0.6MB)

Tool         Duration(s)  Speed(MB/s)  Memory     Status    
----------------------------------------------------------
curl         0.459        1.32         ~10MB      ✅ Success
butterfly-dl 0.612        0.99         ~215MB     ✅ Success
aria2        0.643        0.94         ~50MB      ✅ Success

For very small files, curl's lightweight design provides startup advantages

Medium Files (Luxembourg ~43MB)

Tool         Duration(s)  Speed(MB/s)  Memory     Status    
----------------------------------------------------------
butterfly-dl 3.037        14.07        ~215MB     ✅ Success
aria2        5.447        7.84         ~120MB     ✅ Success
curl         9.349        4.57         ~10MB      ✅ Success

butterfly-dl excels with 3x faster than curl and 79% faster than aria2

Key Performance Insights

• 🎯 Sweet Spot: Medium to large files (>10MB) where butterfly-dl delivers 79-200% speed improvements
• 📊 Memory Consistency: Fixed ~215MB usage regardless of file size (vs aria2's scaling memory)
• ⚡ Speed Scaling: 14.07 MB/s on 43MB files vs aria2's 7.84 MB/s and curl's 4.57 MB/s
• 🔧 Smart Strategy: Automatically uses single connection for small files, optimized parallel connections for larger files
• 🏆 Performance Leader: On medium files, butterfly-dl consistently outperforms both aria2 and curl significantly

Benchmark Features

• 🤖 Automatic Tool Detection - Only tests available tools (curl, aria2, butterfly-dl)
• 📋 Comprehensive Metrics - Duration, speed, memory usage, file integrity validation
• 🔒 MD5 Verification - Ensures all tools download identical, uncorrupted files
• 🧹 Clean Testing - Automatic cleanup of temporary benchmark files
• 📈 Fair Comparison - Same network conditions, same target files, same validation

Running Your Own Benchmarks

# Clone and build
git clone https://github.com/butterfly-osm/butterfly-osm
cd butterfly-osm
cargo build --release

# Test with any supported region
./benchmarks/bench.sh <region>

# Examples covering different file sizes
./benchmarks/bench.sh europe/monaco      # Small: ~1MB
./benchmarks/bench.sh europe/luxembourg  # Small: ~2MB  
./benchmarks/bench.sh europe/belgium     # Medium: ~43MB
./benchmarks/bench.sh europe/netherlands # Large: ~580MB
./benchmarks/bench.sh europe/france      # Large: ~3.5GB

Comparison with Alternatives

License

MIT License - see LICENSE file for details.

Contributing

This project follows XP pair programming with human + AI collaboration. See CLAUDE.md for development guidelines.

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

Who

Butterfly Project built by Pierre pierre@warnier.net for the broader OpenStreetMap community.

Butterfly-dl: The optimal tool for downloading large OpenStreetMap datasets efficiently.