DGE: Distributed Graph Execution

What's this?

Conceptually:

• It provides building blocks to build a computational graph, and generates the corresponding rust codes to execute the graph

• The nodes in the graph represent heterogeneous computations

• The edges represent data flow between nodes

• The computational graph is static, meaning it cannot be changed once defined

• The execution is distributed, meaning that the nodes can be executed on different machines

Concretely:

• You define the graph via crate dge-gen in a .rs file, compile & run it to generate the codes (including main.rs, depends on dge-runtime) that when compiled, can be used to execute the graph

• You compile the main.rs to an executable binary, each subcommand in this binary corresponds to a node in the graph.

• You are responsible to launch the binary with appropriate subcommands. (i.e. DGE does not handle the deployment and launching of the binary)

• You can launch arbitrary amounts (>= 1) of instances of the same subcommand, potentially on different machines

• Each edge is backed by exactly one RabbitMQ queue, the nodes are consumers of this queue

• At-least-once delivery is used, meaning that the business code your wrote for the node should be able to handle duplicate messages

An example

The following graph corresponds to the computation (2 * x) * (x * x):

It is generated with code in dge-example/src/main_generate_code.rs:

use dge_gen;

fn main() {
    let mut graph = dge_gen::Graph::new(
        "dge_example::behaviour::accept_failure::accept_failure",
        "dge_example::behaviour::error::Error",
    );
    let start = graph.start("start");
    let fan_out = graph.fan_out(
        start,
        "input",
        "dge_example::behaviour::data::Integer",
        "duplicate_input",
        10
    );

    // ... some code omitted for brevity ...

    graph
        .generate(
            "dge-example/src/generated",
            "dge_example::behaviour::get_rmq_uri",
            "dge_example_work_exchange",
            "dge_example_retry_exchange",
            "retry_",
            "",
        )
        .unwrap()
}

When the above code is compiled and run, a main.rs will be generated, when the main.rs is compiled, you get an executable with these subcommands:

dge-example

USAGE:
    example <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    double                       # when executed, it creates an OS process that doubles the input
    duplicate-input              # send x to node double and square
    init-exchanges-and-queues    # initialize necessary RabbitMQ entities
    multiply                     # creates a node that does multiplication
    square                       # creates a node that calculates the square

These source files are generated for the above graph:

dge-example/src/generated/
├── double.rs
├── duplicate_input.rs
├── graph.dot
├── graph.svg
├── init_exchanges_and_queues.rs
├── main.rs
├── multiply.rs
└── square.rs