SciRS2 Optim - Production Ready v0.1.0-beta.1

[]

Production-Ready optimization algorithms for the SciRS2 scientific computing library. This comprehensive module provides state-of-the-art optimizers, advanced regularization techniques, intelligent learning rate schedulers, and hardware-aware optimization strategies for machine learning and numerical optimization tasks.

🚀 First Beta Release: This is the production-ready first beta version with 338 passing tests, zero warnings, and comprehensive feature coverage.

Features

🔧 Advanced Optimizers (15+ algorithms)

• First-order: SGD, Adam, AdaGrad, RMSProp, AdamW
• State-of-the-art: LAMB, LARS, Lion, RAdam, Lookahead, SAM (Sharpness-Aware Minimization)
• Second-order: LBFGS, Newton methods
• Specialized: SparseAdam, GroupedAdam, parameter-specific optimizers

📊 Comprehensive Regularization (13+ techniques)

• Weight regularization: L1, L2, Elastic Net, Orthogonal, Spectral Normalization
• Activation regularization: Dropout, Spatial Dropout, DropConnect, Activity regularization
• Advanced techniques: Manifold regularization, Label smoothing, MixUp, Stochastic depth, Weight standardization

📈 Intelligent Learning Rate Schedulers (10+ strategies)

• Adaptive: ReduceOnPlateau, Cosine annealing with warm restarts
• Cyclic: Cyclic LR, One-cycle policy
• Advanced: Curriculum learning, Noise injection, Linear warmup with decay

🏗️ Production-Ready Infrastructure

• Unified API: PyTorch-style Parameter wrapper and optimizer factory
• Memory optimization: In-place operations, mixed precision, gradient checkpointing
• Distributed training: Parameter averaging, gradient compression, asynchronous updates
• Hardware-aware: CPU/GPU/TPU/Edge device specific optimizations

🎯 Domain-Specific Strategies

• Computer Vision: Resolution-adaptive, batch norm tuning, augmentation-aware
• Natural Language Processing: Sequence-adaptive, attention-optimized, vocabulary-aware
• Recommendation Systems: Collaborative filtering, matrix factorization, cold start handling
• Time Series: Temporal dependencies, seasonality adaptation, multi-step optimization

🤖 Meta-Learning & Automation

• Hyperparameter optimization: Bayesian optimization, random search, neural optimizers
• Adaptive selection: Automatic optimizer selection based on problem characteristics
• Benchmarking: Comprehensive evaluation suite with visualization tools

Installation

Add the following to your Cargo.toml:

[dependencies]
scirs2-optim = "0.1.0-beta.1"

To enable advanced features:

[dependencies]
# For metrics integration and hyperparameter optimization
scirs2-optim = { version = "0.1.0-beta.1", features = ["metrics_integration"] }

Available Features:

• metrics_integration: Enables integration with scirs2-metrics for advanced hyperparameter tuning
• Default: All core optimization features are enabled by default

Usage

Quick Start - Traditional API

Basic optimization with traditional interface:

use scirs2_optim::{optimizers, regularizers, schedulers};
use scirs2_core::error::CoreResult;
use ndarray::array;

// Optimizer example: Stochastic Gradient Descent
fn sgd_optimizer_example() -> CoreResult<()> {
    // Create parameters
    let mut params = array![1.0, 2.0, 3.0];
    
    // Create gradients (computed elsewhere)
    let grads = array![0.1, 0.2, 0.3];
    
    // Create SGD optimizer with learning rate 0.01
    let mut optimizer = optimizers::sgd::SGD::new(0.01, 0.9, false);
    
    // Update parameters
    optimizer.step(&mut params, &grads)?;
    
    println!("Updated parameters: {:?}", params);
    
    Ok(())
}

// Adam optimizer with a learning rate scheduler
fn adam_with_scheduler_example() -> CoreResult<()> {
    // Create parameters
    let mut params = array![1.0, 2.0, 3.0];
    
    // Create Adam optimizer with default parameters
    let mut optimizer = optimizers::adam::Adam::new(0.001, 0.9, 0.999, 1e-8);
    
    // Create a learning rate scheduler (exponential decay)
    let mut scheduler = schedulers::exponential_decay::ExponentialDecay::new(
        0.001,  // initial learning rate
        0.95,   // decay rate
        100     // decay steps
    )?;
    
    // Training loop (simplified)
    for epoch in 0..1000 {
        // Compute gradients (would normally be from a model)
        let grads = array![0.1, 0.2, 0.3];
        
        // Update learning rate based on epoch
        let lr = scheduler.get_learning_rate(epoch)?;
        optimizer.set_learning_rate(lr);
        
        // Update parameters
        optimizer.step(&mut params, &grads)?;
        
        if epoch % 100 == 0 {
            println!("Epoch {}, LR: {}, Params: {:?}", epoch, lr, params);
        }
    }
    
    Ok(())
}

// Regularization example
fn regularization_example() -> CoreResult<()> {
    // Parameters
    let params = array![1.0, 2.0, 3.0];
    
    // L1 regularization (Lasso)
    let l1_reg = regularizers::l1::L1::new(0.01);
    let l1_penalty = l1_reg.regularization_term(&params)?;
    let l1_grad = l1_reg.gradient(&params)?;
    
    println!("L1 penalty: {}", l1_penalty);
    println!("L1 gradient contribution: {:?}", l1_grad);
    
    // L2 regularization (Ridge)
    let l2_reg = regularizers::l2::L2::new(0.01);
    let l2_penalty = l2_reg.regularization_term(&params)?;
    let l2_grad = l2_reg.gradient(&params)?;
    
    println!("L2 penalty: {}", l2_penalty);
    println!("L2 gradient contribution: {:?}", l2_grad);
    
    // Elastic Net (combination of L1 and L2)
    let elastic_net = regularizers::elastic_net::ElasticNet::new(0.01, 0.5)?;
    let elastic_penalty = elastic_net.regularization_term(&params)?;
    
    println!("Elastic Net penalty: {}", elastic_penalty);
    
    Ok(())
}

Components

🔧 Advanced Optimizers

State-of-the-art optimization algorithms:

use scirs2_optim::optimizers::{
    // Traditional optimizers
    Optimizer, SGD, Adam, AdaGrad, RMSprop, AdamW,
    // State-of-the-art optimizers
    LAMB,           // Layer-wise Adaptive Moments (large batch optimization)
    LARS,           // Layer-wise Adaptive Rate Scaling
    Lion,           // EvoLved Sign Momentum
    RAdam,          // Rectified Adam
    Lookahead,      // Lookahead optimizer wrapper
    SAM,            // Sharpness-Aware Minimization
    LBFGS,          // Limited-memory BFGS
    SparseAdam,     // Adam for sparse gradients
};

📊 Comprehensive Regularization

Advanced regularization techniques:

use scirs2_optim::regularizers::{
    // Weight regularization
    L1, L2, ElasticNet, Orthogonal, SpectralNorm,
    // Activation regularization
    Dropout, SpatialDropout, DropConnect, ActivityRegularizer,
    // Advanced techniques
    ManifoldRegularizer, LabelSmoothing, MixUp, StochasticDepth,
    WeightStandardization, ShakeDrop, EntropyRegularizer,
};

📈 Intelligent Schedulers

Advanced learning rate scheduling:

use scirs2_optim::schedulers::{
    // Adaptive schedulers
    ReduceOnPlateau, CosineAnnealingWarmRestarts,
    // Cyclic schedulers
    CyclicLR, OneCyclePolicy, 
    // Advanced schedulers
    LinearWarmupDecay, CurriculumScheduler, NoiseInjectionScheduler,
    // Traditional schedulers
    ExponentialDecay, StepDecay, CosineAnnealing,
};

🏗️ Production Infrastructure

Enterprise-grade optimization infrastructure:

use scirs2_optim::{
    // Unified API (PyTorch-style)
    Parameter, OptimizerFactory, OptimizerConfig, UnifiedOptimizer,
    // Hardware-aware optimization
    HardwareAwareOptimizer, HardwarePlatform, PerformanceProfiler,
    // Domain-specific strategies
    DomainSpecificSelector, DomainStrategy, OptimizationContext,
    // Memory optimization
    GradientAccumulator, MicroBatchTrainer, MemoryEfficientTrainer,
    // Distributed training
    DistributedCoordinator, ParameterAverager, GradientCompressor,
};

Advanced Features

🤖 Meta-Learning & Hyperparameter Optimization

Automatic hyperparameter tuning and neural optimizers:

use scirs2_optim::{
    HyperparameterOptimizer, MetaOptimizer, NeuralOptimizer,
    AdaptiveOptimizerSelector, OptimizerStatistics,
};

// Automatic optimizer selection based on problem characteristics
let selector = AdaptiveOptimizerSelector::new();
let recommended = selector.recommend_optimizer(&problem_characteristics)?;

// Neural optimizer that learns to optimize
let mut neural_optimizer = NeuralOptimizer::new(
    input_dim: 784,
    hidden_dim: 128,
    learning_rate: 0.001,
)?;

🎯 Domain-Specific Optimization

Specialized strategies for different domains:

use scirs2_optim::{
    DomainSpecificSelector, DomainStrategy, OptimizationContext,
};

// Computer Vision optimization
let cv_strategy = DomainStrategy::ComputerVision {
    resolution_adaptive: true,
    batch_norm_tuning: true,
    augmentation_aware: true,
};

// NLP optimization
let nlp_strategy = DomainStrategy::NaturalLanguage {
    sequence_adaptive: true,
    attention_optimized: true,
    vocab_aware: true,
};

let optimizer = DomainSpecificSelector::create_optimizer(
    &cv_strategy,
    &optimization_context,
)?;

🔧 Hardware-Aware Optimization

Optimization strategies that adapt to hardware:

use scirs2_optim::{
    HardwareAwareOptimizer, HardwarePlatform, PerformanceProfiler,
};

// Define hardware platform
let platform = HardwarePlatform::GPU {
    memory: 11_000_000_000, // 11GB
    compute_units: 68,
    memory_bandwidth: 616.0,
    architecture: GPUArchitecture::Ampere,
};

// Create hardware-aware optimizer
let mut optimizer = HardwareAwareOptimizer::new(
    platform,
    base_optimizer: "adam",
    config,
)?;

📊 Integration with Metrics

The metrics_integration feature provides integration with scirs2-metrics for metric-based optimization:

use scirs2_optim::metrics::{MetricOptimizer, MetricScheduler, MetricBasedReduceOnPlateau};
use scirs2_optim::optimizers::{SGD, Optimizer};

// Create an SGD optimizer guided by metrics
let mut optimizer = MetricOptimizer::new(
    SGD::new(0.01), 
    "accuracy",  // Metric to optimize
    true        // Maximize
);

// Create a metric-guided learning rate scheduler
let mut scheduler = MetricBasedReduceOnPlateau::new(
    0.1,        // Initial learning rate
    0.5,        // Factor to reduce learning rate (0.5 = halve it)
    3,          // Patience - number of epochs with no improvement
    0.001,      // Minimum learning rate
    "val_loss", // Metric name to monitor
    false,      // Maximize? No, we want to minimize loss
);

// During training loop:
for epoch in 0..num_epochs {
    // Train model for one epoch...
    let train_metrics = train_epoch(&model, &train_data);
    
    // Evaluate on validation set
    let val_metrics = evaluate(&model, &val_data);
    
    // Update optimizer with metric value
    optimizer.update_metric(train_metrics.accuracy);
    
    // Update scheduler with validation loss
    let new_lr = scheduler.step_with_metric(val_metrics.loss);
    
    // Apply scheduler to optimizer
    scheduler.apply_to(&mut optimizer);
    
    // Print current learning rate
    println!("Epoch {}: LR = {}", epoch, new_lr);
}

🔍 Advanced Hyperparameter Search

Bayesian optimization and neural architecture search:

use scirs2_optim::{
    HyperparameterOptimizer, AcquisitionFunction, MetaOptimizer,
};

// Bayesian optimization with Gaussian Process
let mut bayesian_optimizer = HyperparameterOptimizer::bayesian(
    search_space,
    AcquisitionFunction::ExpectedImprovement,
    n_initial_samples: 10,
)?;

// Neural architecture search for optimizer design
let mut nas_optimizer = HyperparameterOptimizer::neural_architecture_search(
    architecture_space,
    performance_predictor,
)?;

// Multi-objective optimization
let pareto_front = bayesian_optimizer.multi_objective_search(
    objectives: vec!["accuracy", "inference_speed", "memory_usage"],
    n_trials: 100,
)?;

🚀 Distributed & Memory Optimization

Production-ready distributed training:

use scirs2_optim::{
    DistributedCoordinator, ParameterAverager, GradientCompressor,
    MicroBatchTrainer, GradientAccumulator,
};

// Distributed training coordinator
let mut coordinator = DistributedCoordinator::new(
    world_size: 8,
    rank: 0,
    backend: "nccl",
)?;

// Gradient compression for communication efficiency
let compressor = GradientCompressor::new(
    CompressionStrategy::TopK { k: 0.1 },
    error_feedback: true,
)?;

// Memory-efficient training with gradient accumulation
let mut trainer = MicroBatchTrainer::new(
    micro_batch_size: 4,
    gradient_accumulation_steps: 8,
    mode: AccumulationMode::Mean,
)?;

Combining Optimizers and Regularizers

Example of how to use optimizers with regularizers:

use scirs2_optim::{optimizers::adam::Adam, regularizers::l2::L2};
use ndarray::Array1;

// Create parameters
let mut params = Array1::from_vec(vec![1.0, 2.0, 3.0]);

// Create gradients (computed elsewhere)
let mut grads = Array1::from_vec(vec![0.1, 0.2, 0.3]);

// Create optimizer
let mut optimizer = Adam::new(0.001, 0.9, 0.999, 1e-8);

// Create regularizer
let regularizer = L2::new(0.01);

// Add regularization gradient
let reg_grads = regularizer.gradient(&params).unwrap();
grads += &reg_grads;

// Update parameters
optimizer.step(&mut params, &grads).unwrap();

Custom Learning Rate Schedulers

Creating a custom learning rate scheduler:

use scirs2_optim::schedulers::Scheduler;
use scirs2_core::error::{CoreError, CoreResult};

struct CustomScheduler {
    initial_lr: f64,
}

impl CustomScheduler {
    fn new(initial_lr: f64) -> Self {
        Self { initial_lr }
    }
}

impl Scheduler for CustomScheduler {
    fn get_learning_rate(&mut self, epoch: usize) -> CoreResult<f64> {
        // Custom learning rate schedule
        // Example: square root decay
        Ok(self.initial_lr / (1.0 + epoch as f64).sqrt())
    }
}

Examples

The module includes 30+ production-ready examples:

🎯 Optimizer Examples

• Basic optimizers: SGD, Adam, RMSprop
• Advanced optimizers: LAMB, LARS, Lion, SAM, LBFGS
• Custom optimizer composition and parameter groups

📈 Scheduler Examples

• Cosine annealing with warm restarts
• One-cycle policy for super-convergence
• Curriculum learning and noise injection

🔧 Advanced Workflows

• Memory-efficient training with gradient accumulation
• Hardware-aware optimization strategies
• Domain-specific optimization for different ML tasks
• Hyperparameter search and meta-learning

🚀 Production Examples

• Distributed training with gradient compression
• Mixed precision training workflows
• Benchmarking and performance profiling

Run examples with: cargo run --example <example_name>

Production Status

✅ Ready for Production Use

• 338/338 tests passing
• Zero compiler warnings
• Zero clippy warnings
• Comprehensive documentation
• Extensive example coverage
• Performance benchmarked against industry standards

Roadmap (Post-Alpha)

• GPU acceleration with CUDA/ROCm kernels
• Automatic differentiation integration
• Differential privacy support
• Advanced tensor core optimizations
• Real-time optimization for streaming data

Contributing

See the CONTRIBUTING.md file for contribution guidelines.

License

This project is dual-licensed under:

• MIT License
• Apache License Version 2.0

You can choose to use either license. See the LICENSE file for details.