Shepherd Framework

An scheduler framework that reliably undertakes data transformation and submission of results with perseverance.

Overview

The scheduler framework is designed to:

• Retry Mechanism: Ensure reliable transformation and submission of data by retrying tasks until completion.
• Non-blocking Architecture: Handle compute-heavy tasks efficiently, especially when data transforming units (DTUs) are hosted remotely (e.g., zkVM provers).

Features

• Flexible Integration: Supports various transformation units and consumer endpoints.
• Robustness: Guarantees task completion through persistent retries.
• Scalability: Optimized for compute-heavy workloads.

Requirements

Ensure you have rustup installed. Compile the system using the Rust nightly channel, as it utilizes nightly-only features such as associated_type_defaults.

Testing

PostgreSQL Database test requires docker to be installed on system & may pose intermittent failures if ran parallely. Use:

cargo test database::postgresql -- --test-threads=1

Documentation

Generate and view the documentation:

cargo doc --open

License

This project is licensed under the terms of the MIT License.