rsrq

Rust Redis queue

rsrq is a minimal Redis-backed queue system written in Rust and inspired by RQ.

Overview:

• Queues are populated by a text file containing commands to be executed (each on a new line).
• Workers can be spawned in parallel to process the queue with various modes.
• Native integration with Snakemake cluster profiles.

Feedback and contributions are welcome!

⚙ Installation

Requires Rust 1.70+

# Crates.io
cargo install rsrq

# or...

# Bioconda
conda install -c conda-forge -c bioconda rsrq

If the command rsrq cannot be found after installation, follow these details.

🖥 CLI usage

This is not an exhaustive list of commands, help can be viewed by running the rsrq --help command.

# Export the Redis connection string
export REDIS_URL=redis://:your-password@your-endpoint-url

# Enqueue commands in "/tmp/cmds.txt" to the "test" queue.
rsrq enqueue test /tmp/cmds.txt

# Spawn 10 workers to process the "test" queue.
rsrq worker test --workers 10

# Check the status
rsrq status

# Purge all information from the redis database
rsrq purge all

🐍 Snakemake

When using Snakemake integration, a cluster profile will need to be created to map the commands for submit, status, and cancel. You are responsible for starting workers that will process the queue(s).

# Export the Snakemake profile
rsrq snakemake config /path/to/directory

# Export the Redis connection string
export REDIS_URL=redis://:your-password@your-endpoint-url

# Run snakemake with the cluster profile
snakemake --profile /path/to/directory

You can specify a queue for each rule using the resources.queue attribute. If none is provided default will be used, e.g.:

rule foo:
    resources:
        queue='something'

Note: Workers do not check the CPU/memory usage of the server, it's entirely up to the user to decide how many workers can run concurrently.