MockForge Chaos

Chaos engineering features for MockForge - fault injection, resilience testing, and advanced orchestration capabilities.

This crate provides comprehensive chaos engineering tools including traffic shaping, fault injection, circuit breakers, bulkheads, and intelligent scenario orchestration. It's designed for testing system resilience, implementing chaos engineering practices, and ensuring application reliability under adverse conditions.

Features

• Fault Injection: Simulate network failures, service outages, and system errors
• Traffic Shaping: Control bandwidth, latency, and packet loss
• Circuit Breakers: Automatic failure detection and recovery
• Bulkheads: Resource isolation and failure containment
• Scenario Orchestration: Complex chaos experiment orchestration
• A/B Testing: Statistical comparison of system behaviors
• Machine Learning: Anomaly detection and predictive remediation
• Multi-Cluster: Distributed chaos testing across clusters
• Observability: Comprehensive metrics and monitoring
• GitOps Integration: Version control for chaos experiments

Quick Start

Basic Fault Injection

use mockforge_chaos::{FaultInjector, FaultType, ChaosConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create fault injector
    let config = ChaosConfig {
        fault_injection: Some(FaultInjectionConfig {
            global_error_rate: 0.1, // 10% error rate
            default_status_codes: vec![500, 502, 503],
            ..Default::default()
        }),
        ..Default::default()
    };

    let injector = FaultInjector::new(config);

    // Inject faults into your service
    let result = injector.inject_fault().await?;
    match result {
        FaultType::HttpError { status_code, .. } => {
            println!("Injected HTTP error: {}", status_code);
        }
        FaultType::NetworkTimeout { .. } => {
            println!("Injected network timeout");
        }
        _ => println!("Other fault injected"),
    }

    Ok(())
}

Circuit Breaker Pattern

use mockforge_chaos::resilience::{CircuitBreaker, CircuitBreakerManager};

let circuit_breaker = CircuitBreaker::new(
    "api_service",
    5,  // failure threshold
    std::time::Duration::from_secs(60), // recovery timeout
);

// Use in your service calls
match circuit_breaker.call(|| async {
    // Your service call here
    reqwest::get("http://api.example.com").await
}).await {
    Ok(response) => println!("Success: {:?}", response),
    Err(e) => println!("Circuit breaker error: {:?}", e),
}

Traffic Shaping

use mockforge_chaos::{TrafficShaper, TrafficShapingConfig};

let config = TrafficShapingConfig {
    bandwidth_limit: Some(1024 * 1024), // 1MB/s
    latency_ms: Some(100),              // 100ms delay
    packet_loss_rate: Some(0.05),       // 5% packet loss
    ..Default::default()
};

let shaper = TrafficShaper::new(config);

// Apply traffic shaping to network interface
shaper.apply_to_interface("eth0").await?;

Core Components

Fault Injection

Simulate various types of failures:

use mockforge_chaos::fault::{FaultInjector, FaultType};

let injector = FaultInjector::new(ChaosConfig::default());

// Different fault types
let faults = vec![
    FaultType::HttpError { status_code: 500, message: "Internal Server Error".to_string() },
    FaultType::NetworkTimeout { duration_ms: 5000 },
    FaultType::ConnectionReset,
    FaultType::DataCorruption { corruption_rate: 0.1 },
];

// Inject random faults
for _ in 0..10 {
    if let Some(fault) = injector.should_inject_fault() {
        println!("Injecting fault: {:?}", fault);
        injector.inject_fault_type(fault).await?;
    }
}

Resilience Patterns

Circuit Breaker

use mockforge_chaos::resilience::CircuitBreaker;

let cb = CircuitBreaker::new("service", 3, std::time::Duration::from_secs(30));

let result = cb.call(|| async {
    // Potentially failing operation
    fallible_operation().await
}).await;

match result {
    Ok(value) => println!("Success: {}", value),
    Err(CircuitBreakerError::Open) => println!("Circuit breaker is open"),
    Err(CircuitBreakerError::Rejected) => println!("Request rejected"),
}

Bulkhead

use mockforge_chaos::resilience::{Bulkhead, BulkheadConfig};

let config = BulkheadConfig {
    max_concurrent_calls: 10,
    max_wait_duration: std::time::Duration::from_secs(5),
};

let bulkhead = Bulkhead::new("resource_pool", config);

let result = bulkhead.execute(|| async {
    // Resource-intensive operation
    heavy_computation().await
}).await;

match result {
    Ok(value) => println!("Completed: {}", value),
    Err(BulkheadError::Rejected) => println!("Bulkhead rejected request"),
}

Scenario Orchestration

Create complex chaos experiments:

use mockforge_chaos::scenario_orchestrator::{ScenarioOrchestrator, OrchestratedScenario, ScenarioStep};

let orchestrator = ScenarioOrchestrator::new();

let scenario = OrchestratedScenario {
    name: "api_failure_simulation".to_string(),
    steps: vec![
        ScenarioStep::Delay { duration_ms: 1000 },
        ScenarioStep::InjectFault { fault_type: FaultType::HttpError { status_code: 503, message: "Service Unavailable".to_string() } },
        ScenarioStep::Delay { duration_ms: 2000 },
        ScenarioStep::InjectFault { fault_type: FaultType::NetworkTimeout { duration_ms: 10000 } },
        ScenarioStep::Assert { condition: "response_time > 5000".to_string() },
    ],
};

orchestrator.execute_scenario(scenario).await?;

A/B Testing

Statistical comparison of system behaviors:

use mockforge_chaos::ab_testing::{ABTestingEngine, ABTestConfig, TestVariant};

let engine = ABTestingEngine::new();

let config = ABTestConfig {
    name: "latency_optimization".to_string(),
    variants: vec![
        TestVariant {
            name: "baseline".to_string(),
            weight: 50,
            config: ChaosConfig::default(),
        },
        TestVariant {
            name: "optimized".to_string(),
            weight: 50,
            config: ChaosConfig {
                latency: Some(LatencyConfig {
                    fixed_delay_ms: Some(50),
                    ..Default::default()
                }),
                ..Default::default()
            },
        },
    ],
    duration_minutes: 60,
    success_criteria: vec![/* ... */],
};

let test_id = engine.start_test(config).await?;
println!("Started A/B test: {}", test_id);

Machine Learning Features

Anomaly Detection

use mockforge_chaos::ml_anomaly_detector::{AnomalyDetector, AnomalyDetectorConfig};

let config = AnomalyDetectorConfig {
    sensitivity: 0.8,
    training_window_hours: 24,
    ..Default::default()
};

let detector = AnomalyDetector::new(config);

// Feed metrics data
detector.add_data_point("response_time", 150.0, chrono::Utc::now()).await?;

// Check for anomalies
if let Some(anomaly) = detector.detect_anomaly("response_time").await? {
    println!("Anomaly detected: {:?}", anomaly);
}

Predictive Remediation

use mockforge_chaos::predictive_remediation::PredictiveRemediationEngine;

let engine = PredictiveRemediationEngine::new();

// Analyze system metrics
let prediction = engine.predict_failure("api_service", 30).await?; // 30-minute prediction

if prediction.probability > 0.8 {
    println!("High failure probability detected!");
    // Trigger remediation
    engine.trigger_remediation(prediction).await?;
}

Advanced Features

Multi-Cluster Orchestration

use mockforge_chaos::multi_cluster::{MultiClusterOrchestrator, ClusterTarget};

let orchestrator = MultiClusterOrchestrator::new();

let targets = vec![
    ClusterTarget {
        name: "production-us-east".to_string(),
        endpoint: "https://k8s-us-east.example.com".to_string(),
        credentials: /* ... */,
    },
    ClusterTarget {
        name: "production-us-west".to_string(),
        endpoint: "https://k8s-us-west.example.com".to_string(),
        credentials: /* ... */,
    },
];

// Execute chaos scenario across clusters
orchestrator.execute_multi_cluster(scenario, targets).await?;

GitOps Integration

use mockforge_chaos::gitops::{GitOpsManager, GitOpsConfig};

let config = GitOpsConfig {
    repository_url: "https://github.com/org/chaos-experiments".to_string(),
    branch: "main".to_string(),
    auth: GitOpsAuth::Token { token: "ghp_...".to_string() },
};

let gitops = GitOpsManager::new(config);

// Sync chaos configurations from Git
gitops.sync_from_git().await?;

// Push experiment results back to Git
gitops.push_results(experiment_results).await?;

Observability API

use mockforge_chaos::observability_api::create_observability_router;
use axum::Router;

let app = Router::new()
    .merge(create_observability_router())
    .layer(ChaosMiddleware::new(ChaosConfig::default()));

// Routes available:
// GET /metrics - Prometheus metrics
// GET /health - Health check
// GET /chaos/status - Chaos experiment status
// POST /chaos/scenarios - Execute scenarios

Configuration

ChaosConfig

use mockforge_chaos::config::ChaosConfig;

let config = ChaosConfig {
    fault_injection: Some(FaultInjectionConfig {
        global_error_rate: 0.05,
        default_status_codes: vec![500, 502, 503],
        network_failure_rate: 0.01,
        timeout_rate: 0.02,
    }),
    latency: Some(LatencyConfig {
        fixed_delay_ms: Some(100),
        jitter_ms: Some(50),
        distribution: LatencyDistribution::Normal,
    }),
    traffic_shaping: Some(TrafficShapingConfig {
        bandwidth_limit: Some(1024 * 1024), // 1MB/s
        packet_loss_rate: Some(0.001),      // 0.1%
    }),
    rate_limiting: Some(RateLimitConfig {
        requests_per_second: 100,
        burst_size: 20,
    }),
    circuit_breaker: Some(CircuitBreakerConfig {
        failure_threshold: 5,
        recovery_timeout_secs: 60,
        success_threshold: 3,
    }),
    bulkhead: Some(BulkheadConfig {
        max_concurrent_calls: 10,
        max_wait_duration: std::time::Duration::from_secs(5),
    }),
};

Integration Examples

With HTTP Services

use mockforge_chaos::middleware::chaos_middleware;
use axum::Router;

let app = Router::new()
    .route("/api/*path", axum::routing::get(handler))
    .layer(chaos_middleware(ChaosConfig::default()));

With gRPC Services

use mockforge_chaos::protocols::grpc::GrpcChaos;

let grpc_chaos = GrpcChaos::new(ChaosConfig::default());

// Apply chaos to gRPC calls
let result = grpc_chaos.intercept_call(request).await?;

With WebSocket Connections

use mockforge_chaos::protocols::websocket::WebSocketChaos;

let ws_chaos = WebSocketChaos::new(ChaosConfig::default());

// Inject chaos into WebSocket messages
ws_chaos.intercept_message(&mut message).await?;

Performance Considerations

• Resource Usage: Chaos features add overhead - monitor system resources
• Distributed Mode: Use Redis backend for multi-instance coordination
• Metrics Collection: Enable metrics for observability without impacting performance
• Gradual Rollout: Start with low fault injection rates and increase gradually

Safety Features

• Approval Workflows: Require approval for destructive chaos experiments
• Automatic Rollback: Built-in remediation for failed experiments
• Safety Checks: Pre-flight validation before executing chaos
• Rate Limiting: Prevent overwhelming systems with too much chaos

Examples

See the examples directory for complete working examples including:

• Basic fault injection scenarios
• Circuit breaker implementations
• Multi-cluster chaos orchestration
• A/B testing setups
• Machine learning integration

Related Crates

• mockforge-core: Core mocking functionality
• mockforge-observability: Metrics and monitoring

License

Licensed under MIT OR Apache-2.0