⚡ Lightning Log

Ultra-fast zero-allocation logging for high-frequency trading and low-latency systems

🚀 Performance Results

Comprehensive benchmarks show Lightning Log's exceptional performance for real-world logging scenarios:

Real-World Performance (Both Libraries Doing Actual Work)

Critical Path Performance

For High-Frequency Trading (HFT)

Lightning Log is the only viable choice:

• Critical Path: 73ns (can process 137 messages per network packet travel)
• Real Work: 190ns per message with full serialization + async processing
• Throughput: 5.26M messages/second
• Consistency: Lock-free, predictable latency
• Production Ready: Graceful shutdown, file output, configuration

The log crate is completely unsuitable for HFT:

• Even with optimal subscriber: 45,381ns (238x slower)
• Variable latency depending on subscriber implementation
• Not designed for sub-microsecond requirements
• Blocking I/O can cause latency spikes

Benchmark Details

# Run comprehensive benchmarks
cargo bench

# Run specific benchmark groups
cargo bench --bench logging_benchmark -- logging_comparison
cargo bench --bench logging_benchmark -- high_throughput

# View detailed results
open target/criterion/report/index.html

# Run fair comparison example
cargo run --example log_crate_comparison

Key Insight: Lightning Log processes 238x more messages in the same time as the log crate, making it the only choice for latency-critical applications like HFT.

📊 Key Features

• ⚡ Sub-100ns latency in the critical path
• 🚫 Zero allocations during logging calls
• 🔄 Binary serialization for maximum performance
• 🔒 Lock-free communication between threads
• 📍 CPU affinity support for logging thread
• 🏷️ Structured logging with message IDs
• ⚙️ Compile-time optimizations via macros
• 📁 Configurable output (console, file, multiple destinations)
• 🛡️ Production-ready with graceful shutdown

🏗️ Architecture Overview

Core Components

1. FastSerialize Trait - Zero-copy binary serialization
2. LightningLogger - Main logging engine with async processing
3. OutputWriter - Multi-destination output management
4. LogEntry - Structured log data container
5. Message Formats - Human-readable format registry

Data Flow

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   User Code     │───▶│   LogEntry       │───▶│  OutputWriter   │
│                 │    │   Creation       │    │                 │
│  lightning_!()  │    │   (~73ns)        │    │  File/Console   │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │
                                ▼
                       ┌──────────────────┐
                       │ LightningLogger  │
                       │   (Async Thread) │
                       └──────────────────┘

🚀 Quick Start

Basic Usage

use lightning_log::*;

// Initialize the logger (call once at startup)
init_lightning_log()?;

// Register message formats (optional, for human-readable output)
register_message_format(1001, "Trade executed: volume={}, price={}, symbol={}".to_string())?;

let volume = 100.0;
let price = 150.25;
let symbol = "AAPL";

// Ultra-fast logging (typically <100ns)
lightning_info!(1001, volume, price, symbol);

Advanced Configuration

use lightning_log::*;
use std::path::PathBuf;

// Configure for production use
let config = LoggerConfig {
    channel_capacity: 1_000_000,        // Large buffer for high throughput
    destinations: vec![
        LogDestination::Stdout,         // Console output
        LogDestination::File(PathBuf::from("logs/trading.log")),
    ],
    file_buffer_size: 128 * 1024,       // 128KB buffer
    enable_cpu_affinity: true,          // Pin to specific CPU core
};

init_lightning_log_with_config(config)?;

// Register message formats
register_message_format(1001, "Trade: volume={}, price={}, symbol={}".to_string())?;

// Your trading logic here...
let volume = 1000.0;
let price = 150.25;
let symbol = "AAPL";
lightning_info!(1001, volume, price, symbol);

// Graceful shutdown (important for production)
shutdown_lightning_log();

📁 Project Structure

lightning-log/
├── src/
│   └── lib.rs              # Main library implementation
├── benches/
│   └── logging_benchmark.rs # Comprehensive performance benchmarks
├── examples/
│   ├── basic_usage.rs      # Basic usage example
│   └── trading_simulation.rs # High-frequency trading simulation
├── target/
│   └── criterion/          # Benchmark results and reports
└── README.md              # This file

🔧 Installation

Add to your Cargo.toml:

[dependencies]
lightning-log = "1.1.0"

For development and benchmarking:

[dev-dependencies]
criterion = "0.5"
rand = "0.8"

📚 API Reference

Initialization Functions

• init_lightning_log() - Initialize with default configuration
• init_lightning_log_with_capacity(capacity: usize) - Initialize with custom channel capacity
• init_lightning_log_with_config(config: LoggerConfig) - Initialize with full configuration

Logging Macros

• lightning_debug!(message_id, args...) - Debug level logging
• lightning_info!(message_id, args...) - Info level logging
• lightning_warn!(message_id, args...) - Warning level logging
• lightning_error!(message_id, args...) - Error level logging

Configuration

• LoggerConfig - Main configuration structure
• LogDestination - Output destination (Stdout, Stderr, File, Multiple)
• register_message_format(id, format) - Register human-readable format strings

Shutdown

• shutdown_lightning_log() - Initiate graceful shutdown
• get_global_logger() - Get reference to logger for manual control

🧪 Running Benchmarks

# Run all benchmarks
cargo bench

# Run specific benchmark group
cargo bench --bench logging_benchmark -- logging_comparison

# Generate HTML reports
cargo bench --bench logging_benchmark

# View benchmark results
open target/criterion/report/index.html

🎯 Use Cases

High-Frequency Trading (HFT)

• Order execution logging
• Market data capture
• Risk management alerts
• Performance monitoring

Low-Latency Systems

• Real-time data processing
• Financial market infrastructure
• Gaming servers
• Telecommunications

General Applications

• Performance-sensitive logging
• Large-scale data processing
• Real-time analytics
• System monitoring

🔍 Benchmark Details

Test Scenarios

1. Logging Comparison - Lightning Log vs println! vs log crate
2. Data Types - Performance across different data types
3. Message Complexity - Impact of argument count on performance
4. High Throughput - Bulk message processing performance
5. Latency Distribution - Statistical analysis of latency
6. Memory Patterns - Static vs dynamic string performance
7. Concurrent Logging - Multi-threaded performance

Key Findings

• 156x faster than standard println! for equivalent functionality
• Consistent sub-100ns latency under load
• Zero-allocation guarantee in critical path
• Excellent concurrency scaling with lock-free design
• Predictable latency distribution (no GC pauses)

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Development Setup

# Clone the repository
git clone https://github.com/simplysabir/lightning-log.git
cd lightning-log

# Run tests
cargo test

# Run benchmarks
cargo bench

# Run examples
cargo run --example basic_usage
cargo run --example trading_simulation
cargo run --example log_crate_comparison

📄 License

Licensed under :

• MIT License (LICENSE-MIT)

at your option.

🔬 Research & References

This implementation is based on the paper:

• "Fast Logging for HFT in Rust"
• Focuses on minimizing latency through binary serialization and async processing
• Eliminates memory allocations in the critical path
• Uses lock-free data structures for thread safety

📈 Performance Optimization Techniques

1. Binary Serialization - Custom FastSerialize trait for zero-copy data conversion
2. Async Processing - Separate thread for I/O operations
3. Lock-Free Channels - Crossbeam channels for inter-thread communication
4. CPU Affinity - Pin logging thread to specific CPU cores
5. Buffer Management - Pre-allocated buffers and batch processing
6. Compile-Time Optimization - Macro-based format string processing

⚠️ Important Notes

• Initialization Required - Call init_lightning_log() before logging
• Message Format Registration - Register formats before using message IDs
• Graceful Shutdown - Use shutdown_lightning_log() for clean termination
• Buffer Sizing - Configure appropriate channel capacity for your use case
• Performance vs Functionality - This is optimized for speed, not feature completeness

📞 Support

For questions or support:

• Open an issue on GitHub
• Check the examples and documentation
• Review the benchmark results for performance guidance

Built with ❤️ for high-performance Rust applications