"I don't care what they're talking about. All I want is a nice, fat recording."

— Harry Caul, The Conversation (1974)

Union Square A transparent, high-performance proxy for LLM API calls

Comprehensive observability, testing, and optimization for AI-powered applications.

Project Status

🚧 Early Development - This project is in the initial development phase. Core functionality is being implemented.

Overview

Union Square acts as a drop-in proxy between your applications and LLM providers (OpenAI, Anthropic, AWS Bedrock, Google Vertex AI), capturing every interaction for analysis, debugging, and testing. It's designed to add minimal latency (< 5ms) while providing powerful observability features.

Key Features

• Transparent Proxy - Drop-in replacement requiring no code changes
• Multi-Provider Support - OpenAI, Anthropic, AWS Bedrock, Google Vertex AI
• Low Latency - Asynchronous recording with < 5ms overhead
• Test Case Extraction - Convert problematic conversations into automated tests
• Cost & Performance Analytics - Track token usage, latency, and costs
• Streaming Support - Full support for SSE/streaming responses
• Privacy Controls - Configurable recording rules and PII detection
• Self-Hosted - Run in your own infrastructure

Use Cases

For Developers

• Debug LLM interactions with full conversation history
• Extract and run test cases to prevent regressions
• A/B test different models and providers
• Monitor performance and error rates

For Customer Support

• Look up customer sessions to troubleshoot issues
• Flag problematic conversations for engineering review
• Add context and notes to help developers

For Management

• Track AI costs across applications and teams
• Monitor performance metrics and F-scores
• Optimize model selection and usage patterns

Quick Start

Prerequisites

• Rust 1.88+ (or use the Nix development shell which provides the exact version)
• PostgreSQL 14+
• Docker (optional, for containerized deployment)
• Nix (optional, but recommended for consistent development environment)

Installation

# Clone the repository
git clone https://github.com/jwilger/union_square.git
cd union_square

# Option 1: Using Nix (Recommended)
# This provides a consistent development environment with Rust 1.88.0
nix develop

# Option 2: Manual setup
# Ensure you have Rust 1.88+ installed
rustup update

# Start PostgreSQL
docker-compose up -d

# Build the project
cargo build --release

# Run the server
./target/release/union_square

Using the Nix Development Shell

This project includes a Nix flake that provides a complete development environment. The Nix shell includes:

• Rust 1.88.0 - The exact Rust version used by this project
• Cargo and all standard Rust tools (rustc, rustfmt, clippy, etc.)
• Pre-configured environment - All necessary build dependencies

To use the Nix development shell:

# Enter the development shell
nix develop

# All Rust tools are now available
cargo --version  # Should show cargo 1.88.0
rustc --version  # Should show rustc 1.88.0

# Run any cargo commands as usual
cargo build
cargo test
cargo run

The Nix shell ensures all developers and CI environments use identical tool versions, eliminating "works on my machine" issues.

Basic Usage

Replace your LLM API endpoints with Union Square URLs:

- https://api.openai.com/v1/chat/completions
+ https://your-union-square.com/openai/v1/chat/completions

Add session tracking headers:

X-Union-Square-Session-ID: your-session-id
X-Union-Square-Metadata: {"user_id": "12345", "feature": "chat"}

Configuration

Union Square uses a TOML configuration file. See config.example.toml for all options.

[server]
port = 8080
host = "0.0.0.0"

[database]
url = "postgresql://user:pass@localhost/union_square"

[cache]
enabled = true
ttl_seconds = 3600

[privacy]
pii_detection = true
default_recording = true

Architecture

Union Square follows a functional core, imperative shell architecture:

• Proxy Layer - Minimal overhead request forwarding
• Recording Pipeline - Asynchronous capture and storage
• Analysis Engine - Test evaluation and metrics calculation
• Web Interface - Leptos-based reactive UI

Architecture Decision Records

We use Architecture Decision Records (ADRs) to document significant architectural decisions.

• View ADRs: Visit our ADR documentation on GitHub Pages
• Browse locally: See the docs/adr/ directory
• Create new ADR: Run npm run adr:new

Key architectural decisions are documented including technology choices, patterns, and design rationale.

Development

See CONTRIBUTING.md for development guidelines.

Development Environment

We strongly recommend using the Nix development shell for a consistent environment:

# Enter the Nix shell (provides Rust 1.88.0 and all tools)
nix develop

Common Commands

# Format code
cargo fmt

# Run lints
cargo clippy --all-targets -- -D warnings

# Run tests
cargo test

# Type check
cargo check --all-targets

License

This project is licensed under the MIT License - see the LICENSE file for details.

Roadmap

• Initial project setup
• Core proxy functionality
• Basic web interface
• Test case extraction
• Analytics dashboards
• Plugin system for exporters

Contributing

We welcome contributions! Please see our Contributing Guidelines and Code of Conduct.

Security

For security vulnerabilities, please email security@example.com instead of using the issue tracker.