Pawdist - Playwright Awesome Distributor

Pawdist is a high-performance, Rust-based dynamic test distribution system for Playwright that eliminates test duration imbalances caused by Playwright's standard static sharding (--shard) mechanism.

🎯 Problem Statement

Playwright's built-in sharding mechanism pre-assigns tests to workers, which can lead to:

• Uneven load distribution - Some workers finish early while others continue running
• Longer overall test execution time - Total runtime is limited by the slowest shard
• Inefficient resource utilization - CPU cores sitting idle while other workers are overloaded

🏗️ Solution Architecture

Pawdist implements a proven Manager-Worker architecture where:

• Manager (pawdist-manager): Central control point that scans tests, maintains a work queue, and dynamically distributes tests to workers
• Workers (pawdist-worker): Execute tests and continuously request new work from the manager as they complete tasks
• Communication: High-performance gRPC protocol for low-latency manager-worker communication

Key Benefits

✅ Dynamic Load Balancing - Tests are assigned on-demand as workers become available
✅ Optimal Resource Utilization - No idle workers waiting for pre-assigned shards to complete
✅ Faster Overall Execution - Tests finish when the last test completes, not when the slowest shard finishes
✅ Scalable Architecture - Easy to add/remove workers without reconfiguring test distribution
✅ Automatic Shutdown - Manager shuts down automatically when all tests complete and workers finish

🚀 Installation

From crates.io

cargo install pawdist

Build from Source

git clone https://github.com/muhendiskedibey/pawdist.git
cd pawdist
cargo build --release

The binaries will be available at:

• target/release/pawdist-manager
• target/release/pawdist-worker

📖 Usage

1. Start the Manager

# Start with default settings (current directory, chromium browser)
pawdist-manager

# Or specify custom path and browser
pawdist-manager --playwright-project-dir /path/to/your/playwright/project --project firefox

Manager Options

• --playwright-project-dir <PATH> (Optional, default: current directory) - Path to your Playwright project directory
• --project <NAME> (Optional, default: chromium) - Playwright project name (e.g., chromium, firefox, webkit)
• --port <PORT> (Optional, default: 50051) - gRPC server port
• --host <HOST> (Optional, default: 127.0.0.1) - Host to bind the gRPC server

2. Start Workers

Start one or more workers (can be on the same machine or different machines):

pawdist-worker --browser webkit --parallelism 4

Worker Options

• --manager-address <HOST:PORT> (Optional, default: http://127.0.0.1:50051) - Manager gRPC address
• --parallelism <NUMBER> (Optional, default: CPU cores / 2) - Number of parallel test runners per worker
• --browser <TYPE> (Optional, default: chromium) - Browser type: chromium, firefox, or webkit

3. Example Workflow

# Terminal 1: Start the manager (uses current directory and chromium by default)
pawdist-manager

# Terminal 2: Start worker 1
pawdist-worker --browser chromium --parallelism 2

# Terminal 3: Start worker 2 (optional, for more parallelism)
pawdist-worker --browser chromium --parallelism 4 --manager-address http://127.0.0.1:50051

🔧 How It Works

Detailed Workflow

1. Manager Initialization

   • Scans Playwright project using npx playwright test --list --project <name>
   • Builds a dynamic FIFO queue of all discovered tests
   • Starts gRPC server and waits for worker connections

2. Worker Registration

   • Workers connect to manager via gRPC
   • Manager sends project configuration to workers
   • Workers prepare isolated test environments

3. Dynamic Test Environment Setup

   • Each worker starts parallelism number of persistent browser servers
   • Creates temporary Playwright config files with WebSocket endpoints
   • Spawns persistent Node.js runner processes for test execution

4. Dynamic Test Execution Loop

   • Workers continuously request tests from the manager's queue
   • Manager assigns tests on a first-come-first-served basis
   • Workers execute tests and report results back to manager
   • Process continues until queue is empty

5. Automatic Completion & Cleanup

   • When test queue is empty, workers notify manager and terminate gracefully
   • Workers clean up browser servers, runner processes, and temporary files
   • Manager automatically shuts down when all workers have finished (no manual intervention required)
   • Supports both automatic shutdown (tests complete) and manual shutdown (Ctrl+C)

🛠️ Technical Details

Project Structure

pawdist/
├── Cargo.toml              # Rust project configuration
├── build.rs                # gRPC code generation
├── proto/
│   └── pawdist.proto       # gRPC service definitions
└── src/
    ├── main.rs             # CLI entry point
    ├── manager.rs          # Manager implementation
    ├── worker.rs           # Worker implementation
    └── protocol/           # Generated gRPC code
        └── pawdist.rs

gRPC Service Definition

The communication protocol includes:

• RegisterWorker - Worker registration and configuration exchange
• GetNextTest - Dynamic test assignment from queue
• ReportTestResult - Test execution result reporting
• NotifyWorkerShutdown - Worker notifies manager when shutting down (enables automatic manager shutdown)

Temporary Files

Workers create temporary files in the Playwright project directory:

• pawdist.config.{slot}.ts - Playwright config files with WebSocket endpoints
• pawdist.runner.{slot}.mjs - Node.js test runner scripts

All temporary files are automatically cleaned up when workers shut down.