Rialo Aggregator Utils

This crate provides essential utilities for building robust aggregator programs in the Rialo ecosystem. It offers statistical functions, data processing tools, and error handling utilities to simplify the implementation of aggregator programs that process oracle data.

Overview

Rialo Aggregator Utils serves as a foundation library for developers building aggregator programs. It focuses on providing robust statistical methods and data extraction utilities that handle real-world challenges like outliers, malformed data, and varying data formats from multiple oracle sources.

Note the current implementation is focused on median calculation, with plans to extend to other statistical methods in the future. For future extensions ideas, please refer to the Future Extensions section.

Core Features

Median Calculation

The primary utility is a sophisticated median calculation system designed for processing oracle price data:

• Multiple calculation methods for handling even-sized datasets
• Robust error handling with detailed error reporting
• NaN and outlier resistance for reliable results
• Flexible data extraction from Base64-encoded JSON oracle updates

Error Handling

Comprehensive error handling for common oracle data issues:

• Base64 decoding errors
• JSON parsing failures
• Missing data fields
• Non-numeric values
• Detailed error tracking with index mapping

Median Calculation

MedianMethod Enum

Defines how to calculate the median when dealing with an even number of values:

pub enum MedianMethod {
    /// Average the two middle values: (a + b) / 2 (default)
    Average,
    /// Take the lower of the two middle values
    Lower,
    /// Take the upper of the two middle values
    Upper,
}

Median Struct

The core result structure that implements the RialoEvent trait:

#[derive(Clone, Debug, Default, PartialEq, RialoEvent)]
pub struct Median {
    pub median: f64,
    pub sample_size: usize,
}

MedianResult

Comprehensive result type that includes both the calculated median and any errors encountered:

pub struct MedianResult {
    pub median: Median,
    pub errors: Vec<(usize, ValueExtractionError)>,
}

Usage Examples

Basic Median Calculation

use rialo_aggregator_utils::median::{Median, MedianMethod};
use rialo_oracle_processor_interface::state::OracleUpdate;

// Calculate median with default method (Average)
let result = Median::from_with_method("price", &oracle_updates, MedianMethod::Average);

// Access the median value
let median = result.median;
println!("Median price: {}, Sample size: {}", median.median, median.sample_size);

// Check for errors during processing
for (index, error) in result.errors() {
    println!("Error at index {}: {}", index, error);
}

Different Median Methods

// For even-sized datasets, different methods yield different results
let updates = vec![/* oracle updates with values [10, 20, 30, 40] */];

// Average method: (20 + 30) / 2 = 25.0
let avg_result = Median::from_with_method("price", &updates, MedianMethod::Average);

// Lower method: 20.0
let lower_result = Median::from_with_method("price", &updates, MedianMethod::Lower);

// Upper method: 30.0  
let upper_result = Median::from_with_method("price", &updates, MedianMethod::Upper);

Error-Resilient Processing

// Process oracle updates with potential errors
let result = Median::from_with_method("price", &oracle_updates, MedianMethod::Average);

// The median is calculated from valid values only
println!("Calculated median: {}", result.median.median);

// Inspect any errors that occurred
match result.errors().len() {
    0 => println!("All oracle updates processed successfully"),
    n => {
        println!("Encountered {} errors during processing:", n);
        for (index, error) in result.errors() {
            match error {
                ValueExtractionError::DecodeError(_) => {
                    println!("  Oracle {} had invalid Base64 data", index);
                }
                ValueExtractionError::JsonError(_) => {
                    println!("  Oracle {} had malformed JSON", index);
                }
                ValueExtractionError::MissingKey(key) => {
                    println!("  Oracle {} missing key: {}", index, key);
                }
                ValueExtractionError::NotANumber(val) => {
                    println!("  Oracle {} had non-numeric value: {}", index, val);
                }
            }
        }
    }
}

Backward Compatibility

The crate provides convenient From trait implementations for simple usage:

// Direct conversion for simple cases
let median: Median = ("price", &oracle_updates).into();

// Or get the full result with error information
let result: MedianResult = ("price", &oracle_updates).into();

Data Processing Pipeline

The median calculation follows this robust pipeline:

1. Data Extraction: Decode Base64 oracle data and parse JSON
2. Value Extraction: Extract numeric values for the specified key
3. Error Collection: Track failed extractions with detailed error information
4. Statistical Calculation: Compute median using the specified method
5. Result Packaging: Return both the median and error details

Oracle Data Format

The utility expects oracle updates with Base64-encoded JSON data:

{
  "price": 50000.0,
  "timestamp": 1634567890,
  "source": "exchange_a"
}

The extract_value function handles:

• Base64 decoding of the oracle data
• JSON parsing with error recovery
• Type conversion from JSON numbers to f64
• Missing key detection and reporting

Error Types

ValueExtractionError

Comprehensive error types for different failure modes:

pub enum ValueExtractionError {
    /// Base64 decoding failed
    DecodeError(DecodeError),
    /// JSON parsing failed  
    JsonError(serde_json::Error),
    /// Required key missing from JSON
    MissingKey(String),
    /// Value exists but is not numeric
    NotANumber(String),
}

Each error provides detailed context for debugging and monitoring oracle data quality.

Robustness Features

NaN Handling

• Automatic filtering of NaN values from calculations
• Safe comparison functions that handle NaN edge cases
• Error reporting for NaN values in source data

Outlier Resistance

• Median-based approach naturally resistant to extreme outliers
• Configurable methods for different statistical requirements
• Sample size tracking for confidence assessment

Data Quality Monitoring

• Error indexing to identify problematic oracle sources
• Detailed error types for targeted debugging
• Partial result computation when some oracles fail

Integration with Rialo Ecosystem

Event System Integration

The Median struct implements RialoEvent, enabling direct emission:

// Calculate median and emit as event
let result = Median::from_with_method("price", &updates, MedianMethod::Average);
result.median.emit(program_id, &accounts)?;

Aggregator Program Usage

Perfect for use in aggregator programs:

// In your aggregator program
let oracle_report_data = deserialize_oracle_report(&oracle_account)?;
let median_result = Median::from_with_method(
    "price", 
    &oracle_report_data.into_updates(),
    MedianMethod::Average
);

// Use the median in your business logic
let trading_pairs = TradingPairs::from_median_results(median_result);

Future Extensions

This crate is designed to be extensible with additional utilities:

• Other robust statistics (trimmed mean, weighted median, etc.)
• Time-series analysis utilities
• Data validation and quality scoring
• Multi-key aggregation helpers
• Custom aggregation strategies

Testing

The crate includes comprehensive tests covering:

• Different median methods with various data sizes
• Error handling scenarios for all error types
• Edge cases (empty data, single values, NaN handling)
• Performance testing with large datasets
• Data format variations (integers, floats, mixed types)

Dependencies

• base64: Base64 encoding/decoding for oracle data
• rialo_events_core: Core event framework and derive macros
• rialo_oracle_processor_interface: Oracle update data structures
• serde_json: JSON parsing and manipulation
• thiserror: Error handling and derivation

License

Copyright (c) Subzero Labs, Inc.
SPDX-License-Identifier: Apache-2.0