brk_fetcher

Multi-source Bitcoin price data aggregator with automatic fallback between exchanges.

Overview

This crate provides a unified interface for fetching Bitcoin price data from multiple sources including Binance, Kraken, and a custom BRK API. It implements automatic failover between data sources, retry mechanisms, and supports both real-time and historical price queries using blockchain height or date-based lookups.

Key Features:

• Multi-source price aggregation (Binance, Kraken, BRK API)
• Automatic fallback hierarchy with intelligent retry logic
• Historical price queries by blockchain height or date
• Support for both 1-minute and daily OHLC data
• HAR file import for extended historical data coverage
• Built-in caching with BTreeMap storage for performance

Target Use Cases:

• Bitcoin blockchain analyzers requiring accurate historical pricing
• Applications needing resilient price data with multiple fallbacks
• Tools processing large datasets requiring efficient price lookups

Installation

cargo add brk_fetcher

Quick Start

use brk_fetcher::Fetcher;
use brk_structs::{Date, Height, Timestamp};

// Initialize fetcher with exchange APIs enabled
let mut fetcher = Fetcher::import(true, None)?;

// Fetch price by date
let date = Date::from_ymd(2023, 6, 15)?;
let daily_price = fetcher.get_date(date)?;

// Fetch price by blockchain height
let height = Height::new(800000);
let timestamp = Timestamp::from(1684771200u32);
let block_price = fetcher.get_height(height, timestamp, None)?;

println!("Daily OHLC: {:?}", daily_price);
println!("Block OHLC: {:?}", block_price);

API Overview

Core Types

• Fetcher: Main aggregator managing multiple price data sources
• Binance: Binance exchange API client with HAR file support
• Kraken: Kraken exchange API client for OHLC data
• BRK: Custom API client for blockchain-indexed price data

Key Methods

Fetcher::import(exchanges: bool, hars_path: Option<&Path>) -> Result<Self> Creates a new fetcher instance with configurable data sources.

get_date(&mut self, date: Date) -> Result<OHLCCents> Retrieves daily OHLC data for the specified date with automatic source fallback.

get_height(&mut self, height: Height, timestamp: Timestamp, previous_timestamp: Option<Timestamp>) -> Result<OHLCCents> Fetches price data for a specific blockchain height using minute-level precision.

Data Source Hierarchy

1. Kraken API - Primary source for both 1-minute and daily data
2. Binance API - Secondary source with extended HAR file support
3. BRK API - Fallback source using blockchain-indexed pricing data

Error Handling

The fetcher implements aggressive retry logic with exponential backoff, attempting each source up to 12 hours (720 retries) before failing. Failed requests trigger cache clearing and source rotation.

Examples

Basic Price Fetching

use brk_fetcher::Fetcher;
use brk_structs::Date;

let mut fetcher = Fetcher::import(true, None)?;

// Fetch Bitcoin price for a specific date
let date = Date::from_ymd(2021, 1, 1)?;
match fetcher.get_date(date) {
    Ok(ohlc) => println!("BTC price on {}: ${}", date, ohlc.close.to_dollars()),
    Err(e) => eprintln!("Failed to fetch price: {}", e),
}

Historical Data with HAR Files

use brk_fetcher::Fetcher;
use std::path::Path;

// Initialize with HAR file path for extended historical coverage
let har_path = Path::new("./import_data");
let mut fetcher = Fetcher::import(true, Some(har_path))?;

// Fetch minute-level data using HAR file fallback
let height = Height::new(650000);
let timestamp = Timestamp::from(1598918400u32); // August 2020
let price_data = fetcher.get_height(height, timestamp, None)?;

Batch Processing with Caching

use brk_fetcher::Fetcher;

let mut fetcher = Fetcher::import(true, None)?;

// Process multiple heights - caching improves performance
for height in 800000..800100 {
    let timestamp = Timestamp::from(1684771200u32 + (height - 800000) * 600);

    match fetcher.get_height(Height::new(height), timestamp, None) {
        Ok(ohlc) => println!("Height {}: ${:.2}", height, ohlc.close.to_dollars()),
        Err(e) => eprintln!("Error at height {}: {}", height, e),
    }
}

// Clear caches when done
fetcher.clear();

Architecture

Retry Mechanism

The crate implements a sophisticated retry system with:

• Default retry count: 6 attempts with 5-second delays
• Extended retry: Up to 720 attempts (12 hours) for critical operations
• Cache invalidation: Automatic cache clearing between retry attempts
• Exponential backoff: 60-second delays for extended retries

Data Aggregation

Price data is aggregated using OHLC (Open, High, Low, Close) calculations spanning timestamp ranges. The find_height_ohlc function computes accurate OHLC values by scanning time series data between block timestamps.

HAR File Processing

Binance integration supports HTTP Archive (HAR) files for extended historical data coverage, parsing browser network captures to extract additional pricing data beyond API limitations.

Code Analysis Summary

Main Types: Fetcher aggregator with Binance, Kraken, and BRK source implementations
Caching: BTreeMap-based caching for both timestamp and date-indexed price data
Network Layer: Built on minreq HTTP client with automatic JSON parsing
Error Handling: Comprehensive retry logic with source rotation and cache management
Dependencies: Integrates brk_structs for type definitions and brk_error for unified error handling
Architecture: Multi-source aggregation pattern with hierarchical fallback and intelligent caching

This README was generated by Claude Code