ai-gateway

AI Gateway

OpenSource Enterprise AI Gateway built in Rust

Govern, Secure, and Optimize your AI Traffic. LangDB AI Gateway provides unified interface to all LLMs using OpenAI API format. Built with performance and reliability in mind.

Key Features

🚀 High Performance

• Built in Rust for maximum speed and reliability
• Seamless integration with any framework (Langchain, Vercel AI SDK, CrewAI, etc.)
• Integrate with any MCP servers(https://docs.langdb.ai/ai-gateway/features/mcp-support)

📊 Enterprise Ready

• Comprehensive usage analytics and cost tracking
• Rate limiting and cost control
• Advanced routing, load balancing and failover
• Evaluations

🔒 Data Control

• Full ownership of your LLM usage data
• Detailed logging and tracing

Looking for More? Try Our Hosted & Enterprise Solutions

🌟 Hosted Version - Get started in minutes with our fully managed solution

• Zero infrastructure management
• Automatic updates and maintenance
• Pay-as-you-go pricing

💼 Enterprise Version - Enhanced features for large-scale deployments

• Advanced team management and access controls
• Custom security guardrails and compliance features
• Intuitive monitoring dashboard
• Priority support and SLA guarantees
• Custom deployment options

Contact our team to learn more about enterprise solutions.

Getting Started

1. Installation

Choose one of these installation methods:

Using Docker (Recommended)

docker run -it \
    -p 8080:8080 \
    -e LANGDB_KEY=your-langdb-key-here \
    langdb/ai-gateway serve

Using Cargo

Install from crates.io:

export RUSTFLAGS="--cfg tracing_unstable --cfg aws_sdk_unstable" 

cargo install ai-gateway

export LANGDB_KEY=your-langdb-key-here
ai-gateway serve

2. Make Your First Request

Test the gateway with a simple chat completion:

# Chat completion with GPT-4
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'

# Or try Claude
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-opus",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'

Providers

LangDB AI Gateway currently supports the following LLM providers. Find all the available models here.

	Provider
	OpenAI
	Google Gemini
	Anthropic
	DeepSeek
	TogetherAI
	XAI
	Meta ( Provided by Bedrock )
	Cohere ( Provided by Bedrock )
	Mistral ( Provided by Bedrock )

API Endpoints

The gateway provides the following OpenAI-compatible endpoints:

• POST /v1/chat/completions - Chat completions

• GET /v1/models - List available models

• POST /v1/embeddings - Generate embeddings

• POST /v1/images/generations - Generate images

Advanced Configuration

Create a config.yaml file:

providers:
  openai: 
    api_key: "your-openai-key-here"
  anthropic: 
    api_key: "your-anthropic-key-here"
  
  # Supports mustache style variables  
  gemini:
    api_key: {{LANGDB_GEMINI_API_KEY}}

http:
  host: "0.0.0.0"
  port: 8080

Command Line Options

# Run with custom host and port
ai-gateway serve --host 0.0.0.0 --port 3000

# Run with CORS origins
ai-gateway serve --cors-origins "http://localhost:3000,http://example.com"

# Run with rate limiting
ai-gateway serve --rate-hourly 1000

# Run with cost limits
ai-gateway serve --cost-daily 100.0 --cost-monthly 1000.0

# Run with custom database connections
ai-gateway serve --clickhouse-url "clickhouse://localhost:9000"

Using Config File

Download the sample configuration from our repo.

1. Copy the example config file:

curl -sL https://raw.githubusercontent.com/langdb/ai-gateway/main/config.sample.yaml -o config.sample.yaml

cp config.sample.yaml config.yaml

Command line options will override corresponding config file settings when both are specified.

Rate Limiting

Rate limiting helps prevent API abuse by limiting the number of requests within a time window. Configure rate limits using:

# Limit to 1000 requests per hour
ai-gateway serve --rate-hourly 1000

Or in config.yaml:

rate_limit:
  hourly: 1000
  daily: 10000
  monthly: 100000

Cost Control

Cost control helps manage API spending by setting daily, monthly, or total cost limits. Configure cost limits using:

# Set daily and monthly limits
ai-gateway serve \
  --cost-daily 100.0 \
  --cost-monthly 1000.0 \
  --cost-total 5000.0

Or in config.yaml:

cost_control:
  daily: 100.0   # $100 per day
  monthly: 1000.0  # $1000 per month
  total: 5000.0    # $5000 total

When a cost limit is reached, the API will return a 429 response with a message indicating the limit has been exceeded.

When a rate limit is exceeded, the API will return a 429 (Too Many Requests) response.

Dynamic Model Routing

LangDB AI Gateway empowers you to implement sophisticated routing strategies for your LLM requests. By utilizing features such as fallback routing, script-based routing, and latency-based routing, you can optimize your AI traffic to balance cost, speed, and availability.

Here's an example of a dynamic routing configuration:

{
    "model": "router/dynamic",
    "messages": [
        { "role": "system", "content": "You are a helpful assistant." },
        { "role": "user", "content": "What is the formula of a square plot?" }
    ],
    "router": {
        "router": "router",
        "type": "fallback", // Type: fallback/script/optimized/percentage/latency
        "targets": [
            { "model": "openai/gpt-4o-mini", "temperature": 0.9, "max_tokens": 500, "top_p": 0.9 },
            { "model": "deepseek/deepseek-chat", "frequency_penalty": 1, "presence_penalty": 0.6 }
        ]
    },
    "stream": false
}

This configuration demonstrates how you can define multiple targets with specific parameters to ensure your requests are handled by the most suitable models. For more detailed information, explore our routing documentation.

Observability

The gateway supports OpenTelemetry tracing with ClickHouse as the storage backend. All traces are stored in the langdb.traces table.

Setting up Tracing

1. Create the traces table in ClickHouse:

# Create langdb database if it doesn't exist
clickhouse-client --query "CREATE DATABASE IF NOT EXISTS langdb"

# Import the traces table schema
clickhouse-client --query "$(cat sql/traces.sql)"

2. Enable tracing by providing the ClickHouse URL when running the server:

ai-gateway serve --clickhouse-url "clickhouse://localhost:9000"

You can also set the URL in your config.yaml:

clickhouse:
  url: "http://localhost:8123"

Querying Traces

The traces are stored in the langdb.traces table. Here are some example queries:

-- Get recent traces
SELECT 
    trace_id,
    operation_name,
    start_time_us,
    finish_time_us,
    (finish_time_us - start_time_us) as duration_us
FROM langdb.traces
WHERE finish_date >= today() - 1
ORDER BY finish_time_us DESC
LIMIT 10;

Leveraging LangDB APIs directly within Clickhouse

Did you know you can call LangDB APIs directly within ClickHouse? Check out our UDF documentation to learn how to use LLMs in your SQL queries!

Running with Docker Compose

For a complete setup including ClickHouse for analytics and tracing, follow these steps:

1. Start the services using Docker Compose:

docker-compose up -d

This will start:

• ClickHouse server on ports 8123 (HTTP)
• All necessary configurations will be loaded from docker/clickhouse/server/config.d

2. Build and run the gateway:

ai-gateway

The gateway will now be running with full analytics and logging capabilities, storing data in ClickHouse.

Using MCP Tools

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Ping the server using the tool and return the response"}],
    "mcp_servers": [{"server_url": "http://localhost:3004"}]
  }'

Development

To get started with development:

1. Clone the repository
2. Copy config.sample.yaml to config.yaml and configure as needed
3. Run cargo build to compile
4. Run cargo test to run tests

Contributing

We welcome contributions! Please check out our Contributing Guide for guidelines on:

• How to submit issues
• How to submit pull requests
• Code style conventions
• Development workflow
• Testing requirements

Logging

The gateway uses tracing for logging. Set the RUST_LOG environment variable to control log levels:

RUST_LOG=debug cargo run serve    # For detailed logs
RUST_LOG=info cargo run serve   # For standard logs

License

This project is released under the Apache License 2.0. See the license file for more information.