triton-distributed
| Crates.io | triton-distributed |
| lib.rs | triton-distributed |
| version | 0.1.2+cb8ea1b |
| created_at | 2025-01-30 20:13:50.824318+00 |
| updated_at | 2025-02-06 03:21:04.768584+00 |
| description | Distributed GenAI Inference Framework |
| homepage | https://github.com/triton-inference-server/triton_distributed |
| repository | https://github.com/triton-inference-server/triton_distributed |
| max_upload_size | |
| id | 1536809 |
| size | 493,067 |
Alec (alec-flowers)
documentation
README
triton-distributed
Overview
This repository contains the core components of a distributed GenAI inference framework written in Rust.
Features:
- A High level API defined in component.rs to build distributed applications.
- A Distributed runtime to manage the distributed execution of the inference graph.
- Uses NATS for component communication and etcd for service discovery, allowing engines to be distributed across multiple nodes while maintaining a unified processing graph.
- Modular design makes it easy to build inference pipelines by composing reusable components.
Install Rust:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Rust Build
cargo build
Run Tests
cargo test