Neuro-Divergent Training

Comprehensive training infrastructure for neural forecasting models with advanced optimization, loss functions, and training strategies specifically designed for time series forecasting.

🎯 Overview

The neuro-divergent-training crate provides a complete training ecosystem for neural time series forecasting models, featuring modern optimizers, specialized loss functions, adaptive learning rate scheduling, and comprehensive evaluation metrics. This crate seamlessly integrates with the ruv-FANN neural network library to provide production-ready training capabilities.

🚀 Key Features

Advanced Optimizers

• Adam: Adaptive Moment Estimation with bias correction and AMSGrad support
• AdamW: Adam with decoupled weight decay for better regularization
• SGD: Stochastic Gradient Descent with momentum and Nesterov acceleration
• RMSprop: Root Mean Square Propagation with centering option
• ForecastingAdam: Custom Adam variant optimized for temporal patterns with seasonal correction

Specialized Loss Functions

Point Forecasting Losses

• MAE: Mean Absolute Error
• MSE: Mean Squared Error
• RMSE: Root Mean Squared Error
• MAPE: Mean Absolute Percentage Error (with epsilon handling)
• SMAPE: Symmetric Mean Absolute Percentage Error
• MASE: Mean Absolute Scaled Error (for seasonal data)

Probabilistic Forecasting Losses

• NegativeLogLikelihood: For probabilistic models with mean and variance
• PinballLoss: For quantile forecasting
• CRPS: Continuous Ranked Probability Score
• GaussianNLL: Gaussian Negative Log-Likelihood
• PoissonNLL: Poisson Negative Log-Likelihood
• NegativeBinomialNLL: Negative Binomial NLL with dispersion parameter

Robust and Custom Losses

• HuberLoss: Robust to outliers with configurable delta
• QuantileLoss: Multi-quantile regression support
• ScaledLoss: Scale-invariant wrapper for any base loss
• SeasonalLoss: Seasonal-aware loss weighting

Learning Rate Schedulers

• ExponentialScheduler: Exponential decay with optional warmup
• StepScheduler: Step-wise decay with milestones
• CosineScheduler: Cosine annealing with warm restarts
• PlateauScheduler: Reduce on loss plateau with patience
• WarmupScheduler: Linear warmup followed by decay
• CyclicScheduler: Cyclic learning rates (triangular, triangular2, exp_range)
• OneCycleScheduler: One cycle policy for fast convergence
• SeasonalScheduler: Seasonal-aware scheduling for time series

Comprehensive Metrics

Point Forecast Metrics

• MAE, MSE, RMSE, MAPE, SMAPE, MASE
• Median Absolute Error
• R², Pearson and Spearman correlations

Probabilistic and Interval Metrics

• Coverage Probability
• CRPS, Energy Score, Log Score
• Interval Score, Winkler Score
• Calibration Error, Sharpness

📦 Installation

Add this to your Cargo.toml:

[dependencies]
neuro-divergent-training = "0.1.0"

# Optional features
neuro-divergent-training = { version = "0.1.0", features = ["parallel", "simd", "checkpointing"] }

Available Features

• default: std, serde, parallel, logging, simd, checkpointing
• std: Standard library support
• serde: Serialization support for checkpointing
• parallel: Parallel processing with Rayon
• logging: Training progress logging
• simd: SIMD acceleration for performance
• checkpointing: Model and optimizer state checkpointing
• mixed_precision: Mixed precision training support

🏗️ Training Architecture

The training system is built around four core components that work together seamlessly:

use neuro_divergent_training::*;

// 1. Choose an optimizer
let optimizer = Adam::new(0.001, 0.9, 0.999)
    .with_epsilon(1e-8);

// 2. Select a loss function
let loss = MAPELoss::new()
    .with_epsilon(1e-8);

// 3. Configure learning rate scheduling
let scheduler = ExponentialScheduler::new(0.001, 0.95)
    .with_warmup(1000)
    .with_min_lr(1e-6);

// 4. Set up evaluation metrics
let mut metrics = MetricCalculator::new();
metrics.add_metric("mae", Box::new(MAE::new()));
metrics.add_metric("mape", Box::new(MAPE::new()));
metrics.add_metric("r2", Box::new(R2::new()));

📚 Usage Examples

Basic Training Setup

use neuro_divergent_training::*;
use ruv_fann::Network;

// Create training data
let training_data = TrainingData {
    inputs: vec![vec![vec![1.0, 2.0, 3.0]]],  // [batch, sequence, features]
    targets: vec![vec![vec![4.0]]],           // [batch, horizon, outputs]
    exogenous: None,
    static_features: None,
    metadata: vec![TimeSeriesMetadata {
        id: "series_1".to_string(),
        frequency: "1H".to_string(),
        seasonal_periods: vec![24, 168],
        scale: Some(1.0),
    }],
};

// Configure training
let config = TrainingConfig {
    max_epochs: 100,
    batch_size: 32,
    validation_frequency: 5,
    patience: Some(10),
    gradient_clip: Some(1.0),
    mixed_precision: false,
    seed: Some(42),
    device: DeviceConfig::Cpu { num_threads: None },
    checkpoint: CheckpointConfig {
        enabled: true,
        save_frequency: 10,
        keep_best_only: true,
        monitor_metric: "val_loss".to_string(),
        mode: CheckpointMode::Min,
    },
};

// Create trainer with ruv-FANN integration
let loss_adapter = LossAdapter::new(Box::new(MAPELoss::new()));
let trainer = TrainingBridge::new()
    .with_ruv_fann_trainer(Box::new(
        ruv_fann::training::IncrementalBackpropagation::new()
    ))
    .with_loss_adapter(loss_adapter)
    .with_config(config);

Advanced Optimizer Configuration

// Adam with AMSGrad
let adam = Adam::new(0.001, 0.9, 0.999)
    .with_epsilon(1e-8)
    .with_amsgrad(true);

// AdamW with weight decay
let adamw = AdamW::new(0.001, 0.9, 0.999, 0.01)
    .with_epsilon(1e-8);

// SGD with Nesterov momentum
let sgd = SGD::new(0.01)
    .with_momentum(0.9)
    .with_weight_decay(1e-4)
    .with_nesterov(true);

// Forecasting-specific Adam
let forecasting_adam = ForecastingAdam::new(0.001, 0.9, 0.999)
    .with_temporal_momentum(0.1)
    .with_seasonal_correction(true)
    .with_lookback_window(24);

Loss Function Selection

// For point forecasting
let mse_loss = MSELoss::new();
let mae_loss = MAELoss::new();
let mape_loss = MAPELoss::new().with_epsilon(1e-6);

// For probabilistic forecasting
let nll_loss = NegativeLogLikelihoodLoss::new();
let pinball_loss = PinballLoss::new(0.5); // Median quantile

// For robust forecasting
let huber_loss = HuberLoss::new(1.0);
let quantile_loss = QuantileLoss::new(vec![0.1, 0.5, 0.9]);

// Custom seasonal loss
let seasonal_loss = SeasonalLoss::new(
    Loss::MSE(MSELoss::new()),
    vec![1.0, 1.2, 0.8, 1.1], // Seasonal weights
);

// Scale-invariant loss
let scaled_loss = ScaledLoss::new(
    Loss::MAE(MAELoss::new()),
    100.0, // Scale factor
);

Learning Rate Scheduling

// Exponential decay with warmup
let exp_scheduler = ExponentialScheduler::new(0.001, 0.95)
    .with_warmup(1000)
    .with_min_lr(1e-6);

// Cosine annealing with restarts
let cosine_scheduler = CosineScheduler::new(0.001, 1000)
    .with_min_lr(1e-6)
    .with_restarts(2.0);

// Plateau reduction
let plateau_scheduler = PlateauScheduler::new(
    0.001,
    PlateauMode::Min,
    0.5,  // Reduction factor
    10,   // Patience
).with_min_lr(1e-6)
 .with_cooldown(5);

// One cycle policy
let one_cycle = OneCycleScheduler::new(0.01, 1000)
    .with_pct_start(0.3)
    .with_div_factor(25.0)
    .with_final_div_factor(10000.0)
    .with_anneal_strategy(AnnealStrategy::Cos);

// Seasonal scheduling
let seasonal_scheduler = SeasonalScheduler::new(
    SchedulerType::Exponential(exp_scheduler),
    vec![1.0, 1.2, 0.8, 1.1], // Seasonal factors
    24, // Season length
);

Model Training Loop

use neuro_divergent_training::*;

fn train_model(
    network: &mut Network<f32>,
    train_data: &TrainingData<f32>,
    val_data: &TrainingData<f32>,
    config: &TrainingConfig<f32>,
) -> TrainingResult<TrainingResults<f32>> {
    
    // Initialize components
    let mut optimizer = Adam::new(0.001, 0.9, 0.999);
    let loss_fn = MAPELoss::new();
    let mut scheduler = ExponentialScheduler::new(0.001, 0.95);
    
    let mut metrics = MetricCalculator::new();
    metrics.add_metric("mae", Box::new(MAE::new()));
    metrics.add_metric("mape", Box::new(MAPE::new()));
    metrics.add_metric("r2", Box::new(R2::new()));
    
    let mut training_history = Vec::new();
    let mut validation_history = Vec::new();
    let mut best_loss = f32::INFINITY;
    let mut patience_counter = 0;
    
    for epoch in 0..config.max_epochs {
        // Training phase
        let train_loss = train_epoch(
            network,
            train_data,
            &mut optimizer,
            &loss_fn,
            config,
        )?;
        
        // Update learning rate
        let lr = scheduler.step(epoch, Some(train_loss))?;
        optimizer.set_learning_rate(lr);
        
        // Validation phase
        if epoch % config.validation_frequency == 0 {
            let val_metrics = validate_epoch(
                network,
                val_data,
                &loss_fn,
                &metrics,
            )?;
            
            let val_loss = val_metrics.get("mape").copied().unwrap_or(f32::INFINITY);
            
            // Early stopping check
            if let Some(patience) = config.patience {
                if val_loss < best_loss {
                    best_loss = val_loss;
                    patience_counter = 0;
                    // Save best model checkpoint
                } else {
                    patience_counter += 1;
                    if patience_counter >= patience {
                        break;
                    }
                }
            }
            
            validation_history.push(EpochMetrics {
                epoch,
                loss: val_loss,
                learning_rate: lr,
                gradient_norm: None,
                additional_metrics: val_metrics,
            });
        }
        
        training_history.push(EpochMetrics {
            epoch,
            loss: train_loss,
            learning_rate: lr,
            gradient_norm: None,
            additional_metrics: HashMap::new(),
        });
    }
    
    Ok(TrainingResults {
        final_loss: training_history.last().map(|m| m.loss).unwrap_or(0.0),
        best_loss,
        epochs_trained: training_history.len(),
        training_history,
        validation_history,
        early_stopped: patience_counter >= config.patience.unwrap_or(usize::MAX),
        training_time: std::time::Duration::from_secs(0), // Would track actual time
    })
}

fn train_epoch(
    network: &mut Network<f32>,
    data: &TrainingData<f32>,
    optimizer: &mut dyn Optimizer<f32>,
    loss_fn: &dyn LossFunction<f32>,
    config: &TrainingConfig<f32>,
) -> TrainingResult<f32> {
    let mut epoch_loss = 0.0;
    let batch_count = (data.inputs.len() + config.batch_size - 1) / config.batch_size;
    
    for batch_idx in 0..batch_count {
        let start_idx = batch_idx * config.batch_size;
        let end_idx = (start_idx + config.batch_size).min(data.inputs.len());
        
        // Forward pass
        let mut batch_loss = 0.0;
        let mut gradients = Vec::new();
        
        for sample_idx in start_idx..end_idx {
            let input = &data.inputs[sample_idx];
            let target = &data.targets[sample_idx];
            
            // Run network forward pass
            let output = network.run(&input[0])?;
            
            // Calculate loss and gradients
            let loss = loss_fn.forward(&output, &target[0])?;
            let grad = loss_fn.backward(&output, &target[0])?;
            
            batch_loss += loss;
            if gradients.is_empty() {
                gradients = vec![grad];
            } else {
                for (g, new_g) in gradients[0].iter_mut().zip(grad.iter()) {
                    *g += *new_g;
                }
            }
        }
        
        // Average gradients
        let batch_size_f32 = (end_idx - start_idx) as f32;
        batch_loss /= batch_size_f32;
        for g in gradients[0].iter_mut() {
            *g /= batch_size_f32;
        }
        
        // Apply gradient clipping if configured
        if let Some(max_norm) = config.gradient_clip {
            optimizer.clip_gradients(&mut gradients, max_norm);
        }
        
        // Optimizer step
        let mut params = vec![network.get_weights()]; // Simplified weight extraction
        optimizer.step(&mut params, &gradients)?;
        network.set_weights(&params[0])?; // Simplified weight setting
        
        epoch_loss += batch_loss;
    }
    
    Ok(epoch_loss / batch_count as f32)
}

fn validate_epoch(
    network: &Network<f32>,
    data: &TrainingData<f32>,
    loss_fn: &dyn LossFunction<f32>,
    metrics: &MetricCalculator<f32>,
) -> TrainingResult<HashMap<String, f32>> {
    let mut predictions = Vec::new();
    let mut targets = Vec::new();
    
    for sample_idx in 0..data.inputs.len() {
        let input = &data.inputs[sample_idx];
        let target = &data.targets[sample_idx];
        
        let output = network.run(&input[0])?;
        predictions.extend(output);
        targets.extend(target[0].clone());
    }
    
    metrics.calculate_all(&targets, &predictions)
}

🔧 Advanced Features

Gradient Clipping and Regularization

// Automatic gradient clipping
let norm = utils::clip_gradients_by_norm(&mut gradients, 1.0);
println!("Gradient norm: {:.4}", norm);

// Weight decay with AdamW
let adamw = AdamW::new(0.001, 0.9, 0.999, 0.01); // 1% weight decay

Mixed Precision Training

let config = TrainingConfig {
    mixed_precision: true,
    // ... other config
};

State Management and Checkpointing

// Save optimizer state
let optimizer_state = optimizer.state();

// Save scheduler state
let scheduler_state = scheduler.state();

// Restore from checkpoint
optimizer.restore_state(optimizer_state)?;
scheduler.restore_state(scheduler_state)?;

Custom Loss Functions

use neuro_divergent_training::*;

struct CustomLoss {
    alpha: f32,
}

impl LossFunction<f32> for CustomLoss {
    fn forward(&self, predictions: &[f32], targets: &[f32]) -> TrainingResult<f32> {
        // Custom loss implementation
        let mse = predictions.iter()
            .zip(targets.iter())
            .map(|(p, t)| (p - t).powi(2))
            .sum::<f32>() / predictions.len() as f32;
        
        Ok(self.alpha * mse)
    }
    
    fn backward(&self, predictions: &[f32], targets: &[f32]) -> TrainingResult<Vec<f32>> {
        // Custom gradient implementation
        let n = predictions.len() as f32;
        let gradients = predictions.iter()
            .zip(targets.iter())
            .map(|(p, t)| 2.0 * self.alpha * (p - t) / n)
            .collect();
        
        Ok(gradients)
    }
    
    fn name(&self) -> &'static str {
        "CustomLoss"
    }
}

📊 Performance Optimization

Memory Efficiency

// Use SIMD for vectorized operations (with simd feature)
#[cfg(feature = "simd")]
use wide::f32x8;

// Memory mapping for large datasets (with checkpointing feature)
#[cfg(feature = "checkpointing")]
use memmap2::Mmap;

Parallel Training

// Enable parallel processing (with parallel feature)
#[cfg(feature = "parallel")]
use rayon::prelude::*;

// Parallel batch processing
batches.par_iter_mut().for_each(|batch| {
    // Process batch in parallel
});

SIMD Acceleration

Enable SIMD features for faster numerical operations:

neuro-divergent-training = { version = "0.1.0", features = ["simd"] }

🔬 Integration with ruv-FANN

The training system seamlessly integrates with ruv-FANN neural networks:

use ruv_fann::{Network, training::*};
use neuro_divergent_training::*;

// Create training bridge
let bridge = TrainingBridge::new()
    .with_ruv_fann_trainer(Box::new(IncrementalBackpropagation::new()))
    .with_loss_adapter(LossAdapter::new(Box::new(MAPELoss::new())));

// Use ruv-FANN error functions
impl ruv_fann::training::ErrorFunction<f32> for LossAdapter<f32> {
    fn calculate(&self, actual: &[f32], desired: &[f32]) -> f32 {
        self.calculate_loss(actual, desired).unwrap_or(0.0)
    }
    
    fn derivative(&self, actual: f32, desired: f32) -> f32 {
        // Gradient calculation for single values
        self.calculate_gradient(&[actual], &[desired])
            .unwrap_or_default()
            .first()
            .copied()
            .unwrap_or(0.0)
    }
}

📈 Monitoring and Logging

Training Progress

#[cfg(feature = "logging")]
use log::{info, debug};

// Log training progress
info!("Epoch {}: train_loss={:.4}, val_loss={:.4}, lr={:.6}", 
      epoch, train_loss, val_loss, learning_rate);

// Debug gradient information
debug!("Gradient norm: {:.4}", gradient_norm);

Metrics Collection

let mut metrics = MetricCalculator::new();
metrics.add_metric("mae", Box::new(MAE::new()));
metrics.add_metric("mse", Box::new(MSE::new()));
metrics.add_metric("r2", Box::new(R2::new()));
metrics.add_metric("mape", Box::new(MAPE::new()));

let results = metrics.calculate_all(&y_true, &y_pred)?;
for (name, value) in results {
    println!("{}: {:.4}", name, value);
}

🎛️ Configuration Examples

Production Training Configuration

let config = TrainingConfig {
    max_epochs: 1000,
    batch_size: 64,
    validation_frequency: 5,
    patience: Some(20),
    gradient_clip: Some(1.0),
    mixed_precision: true,
    seed: Some(42),
    device: DeviceConfig::Cpu { num_threads: Some(8) },
    checkpoint: CheckpointConfig {
        enabled: true,
        save_frequency: 50,
        keep_best_only: true,
        monitor_metric: "val_mape".to_string(),
        mode: CheckpointMode::Min,
    },
};

Development/Debug Configuration

let config = TrainingConfig {
    max_epochs: 10,
    batch_size: 8,
    validation_frequency: 1,
    patience: None,
    gradient_clip: Some(10.0),
    mixed_precision: false,
    seed: Some(42),
    device: DeviceConfig::Cpu { num_threads: Some(1) },
    checkpoint: CheckpointConfig {
        enabled: false,
        save_frequency: 1,
        keep_best_only: false,
        monitor_metric: "val_loss".to_string(),
        mode: CheckpointMode::Min,
    },
};

🔍 Testing and Validation

Unit Tests

The crate includes comprehensive unit tests for all components:

cargo test

Property-Based Testing

use proptest::prelude::*;

proptest! {
    #[test]
    fn test_optimizer_convergence(
        learning_rate in 0.001f32..0.1f32,
        momentum in 0.0f32..0.99f32
    ) {
        let mut optimizer = Adam::new(learning_rate, momentum, 0.999);
        // Test convergence properties
    }
}

Benchmarking

cargo bench

Run benchmarks to measure performance:

use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn benchmark_adam_optimizer(c: &mut Criterion) {
    c.bench_function("adam_step", |b| {
        let mut optimizer = Adam::new(0.001, 0.9, 0.999);
        let mut params = vec![vec![1.0; 1000]];
        let gradients = vec![vec![0.1; 1000]];
        
        b.iter(|| {
            optimizer.step(black_box(&mut params), black_box(&gradients)).unwrap();
        });
    });
}

criterion_group!(benches, benchmark_adam_optimizer);
criterion_main!(benches);

📖 API Documentation

Core Traits

• Optimizer<T>: Optimization algorithms interface
• LossFunction<T>: Loss function interface
• LearningRateScheduler<T>: Learning rate scheduling interface
• Metric<T>: Evaluation metrics interface

Builder Patterns

• OptimizerBuilder<T>: Fluent optimizer construction
• SchedulerBuilder<T>: Fluent scheduler construction

Error Handling

All operations return TrainingResult<T> for comprehensive error handling:

pub type TrainingResult<T> = Result<T, TrainingError>;

#[derive(Error, Debug)]
pub enum TrainingError {
    InvalidConfig(String),
    DataError(String),
    OptimizerError(String),
    LossError(String),
    // ... other error types
}

🚀 Performance Benchmarks

Optimizer Performance

Loss Function Performance

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

git clone https://github.com/ruvnet/ruv-FANN.git
cd ruv-FANN/neuro-divergent/neuro-divergent-training
cargo build --all-features
cargo test --all-features

📄 License

This project is licensed under the MIT OR Apache-2.0 license.

🔗 Related Projects

• ruv-FANN: Fast Artificial Neural Network library
• neuro-divergent-models: Forecasting model architectures
• neuro-divergent-data: Data processing utilities

Built with ❤️ for the time series forecasting community