📖 Overview

LLM Test Bench is a powerful, enterprise-grade framework built in Rust for comprehensive testing, benchmarking, and evaluation of Large Language Models. It provides a unified interface to test multiple LLM providers, evaluate responses with sophisticated metrics, and visualize results through an intuitive dashboard.

Why LLM Test Bench?

• 🚀 Multi-Provider Support: Test 14+ LLM providers with 65 models through a single, unified interface
• 🆕 Latest Models: Full support for GPT-5, Claude Opus 4, Gemini 2.5, and all 2025 releases
• 📊 Comprehensive Metrics: Evaluate models with perplexity, coherence, relevance, faithfulness, and custom evaluators
• ⚡ High Performance: Built in Rust for speed, safety, and scalability
• 🎨 Rich Visualization: Interactive dashboards with real-time metrics and beautiful charts
• 🔌 Extensible: Plugin system, custom evaluators, and distributed computing support
• 🐳 Production Ready: Docker support, monitoring, REST/GraphQL APIs, and WebSocket streaming

✨ Features

Core Capabilities

🤖 Multi-Provider LLM Support

OpenAI (27 models)

gpt-5
gpt-4.5, gpt-4.5-2025-02-27
gpt-4.1, gpt-4.1-2025-04
gpt-4o, gpt-4o-2024-11-20, gpt-4o-2024-08-06, gpt-4o-2024-05-13
gpt-4o-mini, gpt-4o-mini-2024-07-18
o1, o1-preview, o1-preview-2024-09-12, o1-mini, o1-mini-2024-09-12, o3-mini
gpt-4-turbo, gpt-4-turbo-2024-04-09, gpt-4-turbo-preview
gpt-4-0125-preview, gpt-4-1106-preview
gpt-4, gpt-4-0613
gpt-3.5-turbo, gpt-3.5-turbo-0125, gpt-3.5-turbo-1106

Anthropic (15 models)

claude-opus-4, claude-opus-4-20250501
claude-sonnet-4.5, claude-sonnet-4.5-20250901
claude-sonnet-4, claude-sonnet-4-20250514
claude-3-5-sonnet-latest, claude-3-5-sonnet-20241022, claude-3-5-sonnet-20240620
claude-3-5-haiku-latest, claude-3-5-haiku-20241022
claude-3-opus-latest, claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307

Google Gemini (16 models)

gemini-2.5-pro
gemini-2.5-computer-use, gemini-2.5-computer-use-20251007
gemini-2.0-flash-exp, gemini-2.0-flash-thinking-exp-1219
gemini-1.5-pro, gemini-1.5-pro-latest, gemini-1.5-pro-002, gemini-1.5-pro-001
gemini-1.5-flash, gemini-1.5-flash-latest, gemini-1.5-flash-002
gemini-1.5-flash-001, gemini-1.5-flash-8b
gemini-pro, gemini-pro-vision

Mistral AI (7 models)

mistral-code, mistral-code-20250604
magistral-large, magistral-medium, magistral-small
voxtral-small, voxtral-small-20250701

Additional Providers

• Azure OpenAI: All OpenAI models via Azure endpoints
• AWS Bedrock: Claude, Llama, Titan, and more
• Cohere: Command, Command R/R+
• Open Source: Ollama, Hugging Face, Together AI, Replicate
• Specialized: Groq, Perplexity AI

📈 Advanced Evaluation Metrics

• Perplexity Analysis: Statistical language model evaluation
• Coherence Scoring: Semantic consistency and logical flow
• Relevance Evaluation: Context-aware response quality
• Faithfulness Testing: Source attribution and hallucination detection
• LLM-as-Judge: Use LLMs to evaluate other LLMs
• Text Analysis: Readability, sentiment, toxicity, PII detection
• Custom Evaluators: Build your own evaluation metrics

🎯 Benchmarking & Testing

• Systematic Testing: Automated test suites with rich assertions
• Comparative Analysis: Side-by-side model comparison
• Performance Profiling: Latency, throughput, and cost tracking
• A/B Testing: Statistical significance testing for model selection
• Optimization Tools: Automatic parameter tuning and model recommendation

📊 Visualization & Reporting

• Interactive Dashboard: Real-time metrics with Chart.js
• Rich Charts: Performance graphs, cost analysis, trend visualization
• Multiple Formats: HTML reports, JSON exports, custom templates
• Cost Analysis: Track spending across providers and models
• Historical Trends: Long-term performance tracking

🌐 API & Integration

• REST API: Complete HTTP API with authentication
• GraphQL: Flexible query interface for complex data needs
• WebSocket: Real-time streaming and live updates
• Monitoring: Prometheus metrics and health checks
• Distributed Computing: Scale benchmarks across multiple nodes

🔌 Extensibility

• Plugin System: WASM-based sandboxed plugins
• Custom Evaluators: Implement domain-specific metrics
• Multimodal Support: Image, audio, and video evaluation
• Database Backend: PostgreSQL with repository pattern
• Flexible Architecture: Clean, modular design for easy extension

🚀 Quick Start

Prerequisites

• Rust: 1.75.0 or later (Install Rust)
• API Keys: At least one LLM provider API key

Installation

# Clone the repository
git clone https://github.com/globalbusinessadvisors/llm-test-bench.git
cd llm-test-bench

# Build the project
cargo build --release

# Install CLI globally (optional)
cargo install --path cli

Configuration

Set up your API keys as environment variables:

# OpenAI
export OPENAI_API_KEY="sk-..."

# Anthropic
export ANTHROPIC_API_KEY="sk-ant-..."

# Google
export GOOGLE_API_KEY="..."

# AWS Bedrock
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_REGION="us-east-1"

Or create a .env file:

cp .env.example .env
# Edit .env with your API keys

Basic Usage

# Run a simple benchmark with GPT-5
llm-test-bench bench --provider openai --model gpt-5 --prompt "Explain quantum computing"

# Test with Claude Opus 4
llm-test-bench bench --provider anthropic --model claude-opus-4 --prompt "Code review this function"

# Use Gemini 2.5 Computer Use
llm-test-bench bench --provider google --model gemini-2.5-computer-use --prompt "Automate this task"

# Compare multiple models across providers
llm-test-bench compare \
  --models "openai:gpt-5,anthropic:claude-opus-4,google:gemini-2.5-pro" \
  --prompt "Write a Python function to sort a list"

# Benchmark code models
llm-test-bench bench --provider mistral --model mistral-code --prompt "Implement binary search"

# Analyze results
llm-test-bench analyze --results benchmark_results.json

# Launch interactive dashboard
llm-test-bench dashboard --port 8080

# Optimize model selection
llm-test-bench optimize \
  --metric latency \
  --max-cost 0.01 \
  --dataset prompts.json

Docker Deployment

# Using Docker Compose (includes PostgreSQL, Redis, Prometheus)
docker-compose up -d

# Access the dashboard
open http://localhost:8080

# View metrics
open http://localhost:9090  # Prometheus

📚 Documentation

Getting Started

• Quick Start Guide - Get up and running in 5 minutes
• CLI Reference - Complete command-line documentation
• Configuration Guide - Advanced configuration options

Architecture & Design

• Architecture Overview - System design and components
• Workspace Structure - Project organization
• Technical Architecture - Deep dive into design

Features

• Provider Support - All supported LLM providers
• API Documentation - REST & GraphQL API reference
• Monitoring - Observability and metrics
• Distributed Computing - Scaling across nodes
• Multimodal - Image, audio, and video support
• Plugins - Extensibility and custom plugins

Deployment

• Docker Deployment - Containerized deployment guide
• Database Setup - PostgreSQL configuration

Development

• Phase Implementation Reports - Detailed implementation history
• Contributing Guide - How to contribute
• Development Setup - Set up your dev environment

🏗️ Architecture

LLM Test Bench follows a clean, modular architecture:

┌─────────────────────────────────────────────────────────────┐
│                        CLI Layer                            │
│  bench │ compare │ analyze │ dashboard │ optimize          │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                     Core Library (core/)                    │
├─────────────────────────────────────────────────────────────┤
│  • Providers      • Evaluators     • Orchestration          │
│  • Analytics      • Visualization  • Monitoring             │
│  • Distributed    • Plugins        • Multimodal             │
│  • API Server     • Database       • Configuration          │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                    External Services                        │
│  LLM APIs │ PostgreSQL │ Redis │ Prometheus │ S3            │
└─────────────────────────────────────────────────────────────┘

Key Components

• Providers: Unified interface for 14+ LLM providers
• Evaluators: Pluggable metrics for response quality assessment
• Orchestration: Intelligent routing, ranking, and comparison
• Visualization: Interactive dashboards and rich reporting
• API Server: REST, GraphQL, and WebSocket endpoints
• Distributed: Cluster coordination for large-scale benchmarks
• Monitoring: Prometheus metrics and health checks
• Plugins: WASM-based extensibility system

🛠️ Technology Stack

• Language: Rust 🦀
• CLI: Clap (command-line parsing)
• Async: Tokio (async runtime)
• HTTP: Axum (web framework)
• Database: SQLx + PostgreSQL
• Serialization: Serde (JSON/YAML)
• GraphQL: Async-GraphQL
• Monitoring: Prometheus client
• WebSocket: Tokio-Tungstenite
• Distributed: Custom protocol over TCP
• Plugins: Wasmtime (WASM runtime)

📦 Project Structure

llm-test-bench/
├── cli/                    # Command-line interface
│   ├── src/
│   │   ├── commands/      # CLI commands (bench, compare, etc.)
│   │   └── main.rs
│   └── tests/             # Integration tests
├── core/                   # Core library
│   ├── src/
│   │   ├── providers/     # LLM provider implementations
│   │   ├── evaluators/    # Evaluation metrics
│   │   ├── orchestration/ # Model routing & comparison
│   │   ├── visualization/ # Dashboard & charts
│   │   ├── api/           # REST/GraphQL/WebSocket
│   │   ├── distributed/   # Cluster coordination
│   │   ├── monitoring/    # Metrics & health checks
│   │   ├── plugins/       # Plugin system
│   │   ├── multimodal/    # Image/audio/video
│   │   ├── analytics/     # Statistics & optimization
│   │   └── config/        # Configuration
│   └── tests/             # Unit & integration tests
├── docs/                   # Documentation
├── examples/               # Usage examples
├── plans/                  # Architecture & planning docs
└── docker-compose.yml      # Docker deployment

🎯 Use Cases

1. Model Selection

Compare multiple LLM providers to choose the best model for your use case based on quality, cost, and latency.

2. Quality Assurance

Systematic testing of LLM applications with rich assertions and automated evaluation metrics.

3. Performance Benchmarking

Measure and track latency, throughput, and cost across different models and configurations.

4. Regression Testing

Ensure model updates don't degrade quality with historical comparison and automated alerts.

5. Cost Optimization

Identify the most cost-effective model that meets your quality requirements.

6. Research & Experimentation

Rapid prototyping and comparison of different prompts, models, and parameters.

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

# Clone and build
git clone https://github.com/globalbusinessadvisors/llm-test-bench.git
cd llm-test-bench
cargo build

# Run tests
cargo test

# Run with logging
RUST_LOG=debug cargo run -- bench --help

# Format code
cargo fmt

# Lint
cargo clippy -- -D warnings

Areas for Contribution

• 🔌 New LLM provider integrations
• 📊 Additional evaluation metrics
• 🎨 Visualization improvements
• 📝 Documentation enhancements
• 🐛 Bug fixes and performance improvements
• ✨ New features and capabilities

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with Rust 🦀
• Inspired by the need for comprehensive LLM testing tools
• Thanks to all contributors and the open-source community

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: docs/

🗺️ Roadmap

Completed ✅

• ✅ Multi-provider LLM support (14+ providers)
• ✅ Advanced evaluation metrics
• ✅ Visualization dashboard
• ✅ REST/GraphQL/WebSocket APIs
• ✅ Distributed computing
• ✅ Monitoring & observability
• ✅ Plugin system
• ✅ Docker deployment
• ✅ PostgreSQL backend

In Progress 🚧

• 🚧 Enhanced multimodal support
• 🚧 Advanced cost optimization
• 🚧 Plugin marketplace
• 🚧 Cloud deployment templates

Planned 📋

• 📋 Real-time collaboration features
• 📋 Advanced A/B testing framework
• 📋 Integration with MLOps platforms
• 📋 Enterprise SSO and RBAC