RuVector ONNX Embeddings

Production-ready ONNX-based embedding generation for semantic search and RAG pipelines in pure Rust

This library provides a complete embedding generation system built entirely in Rust using ONNX Runtime. Designed for high-performance vector databases, semantic search engines, and AI applications.

Table of Contents

• Features
• Quick Start
• Installation
• Supported Models
• Tutorial: Step-by-Step Guide
  • Step 1: Basic Embedding Generation
  • Step 2: Batch Processing
  • Step 3: Building a Semantic Search Engine
  • Step 4: Creating a RAG Pipeline
  • Step 5: Text Clustering
• Configuration Reference
• Pooling Strategies
• Performance Benchmarks
• API Reference
• Architecture
• Troubleshooting

Features

Quick Start

use ruvector_onnx_embeddings::{Embedder, PretrainedModel};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create embedder with default model
    let mut embedder = Embedder::default_model().await?;

    // Generate embedding
    let embedding = embedder.embed_one("Hello, world!")?;
    println!("Embedding dimension: {}", embedding.len()); // 384

    // Compute semantic similarity
    let sim = embedder.similarity(
        "I love programming in Rust",
        "Rust is my favorite language"
    )?;
    println!("Similarity: {:.4}", sim); // ~0.85

    Ok(())
}

Installation

Step 1: Add Dependencies

[dependencies]
ruvector-onnx-embeddings = { path = "examples/onnx-embeddings" }
tokio = { version = "1", features = ["full"] }
anyhow = "1.0"

Step 2: Choose Features (Optional)

Step 3: Run Examples

# Basic example
cargo run --example basic_embedding

# Full demo with all features
cargo run

Supported Models

Model Comparison Table

Model Selection Flowchart

┌─────────────────────────────────────────────────────────────────┐
│                     Which Model Should I Use?                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Priority: Speed?       ──────►  AllMiniLmL6V2 (23MB, 384d)     │
│                                                                  │
│  Priority: Quality?     ──────►  AllMpnetBaseV2 (110MB, 768d)   │
│                                                                  │
│  Building search?       ──────►  BgeSmallEnV15 or E5SmallV2     │
│                                                                  │
│  Multilingual?          ──────►  GteSmall                       │
│                                                                  │
│  Production RAG?        ──────►  BgeBaseEnV15 or E5BaseV2       │
│                                                                  │
│  Memory constrained?    ──────►  AllMiniLmL6V2                  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Tutorial: Step-by-Step Guide

Step 1: Basic Embedding Generation

Goal: Generate your first embedding and understand the output.

use ruvector_onnx_embeddings::{Embedder, EmbedderConfig, PretrainedModel};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Create an embedder (downloads model on first run)
    println!("Loading model...");
    let mut embedder = Embedder::default_model().await?;

    // 2. Check model info
    println!("Model: {}", embedder.model_info().name);
    println!("Dimension: {}", embedder.dimension());
    println!("Max tokens: {}", embedder.max_length());

    // 3. Generate an embedding
    let text = "The quick brown fox jumps over the lazy dog.";
    let embedding = embedder.embed_one(text)?;

    // 4. Examine the output
    println!("\nInput: \"{}\"", text);
    println!("Output shape: [{} dimensions]", embedding.len());
    println!("First 5 values: [{:.4}, {:.4}, {:.4}, {:.4}, {:.4}]",
        embedding[0], embedding[1], embedding[2], embedding[3], embedding[4]);

    // 5. Compute similarity between texts
    let text1 = "I love programming in Rust.";
    let text2 = "Rust is my favorite programming language.";
    let text3 = "The weather is nice today.";

    let sim_related = embedder.similarity(text1, text2)?;
    let sim_unrelated = embedder.similarity(text1, text3)?;

    println!("\nSimilarity comparisons:");
    println!("  \"{}\" vs \"{}\"", text1, text2);
    println!("  Similarity: {:.4} (high - related topics)", sim_related);
    println!();
    println!("  \"{}\" vs \"{}\"", text1, text3);
    println!("  Similarity: {:.4} (low - unrelated topics)", sim_unrelated);

    Ok(())
}

Expected Output:

Loading model...
Model: all-MiniLM-L6-v2
Dimension: 384
Max tokens: 256

Input: "The quick brown fox jumps over the lazy dog."
Output shape: [384 dimensions]
First 5 values: [0.0234, -0.0156, 0.0891, -0.0412, 0.0567]

Similarity comparisons:
  "I love programming in Rust." vs "Rust is my favorite programming language."
  Similarity: 0.8523 (high - related topics)

  "I love programming in Rust." vs "The weather is nice today."
  Similarity: 0.1234 (low - unrelated topics)

Step 2: Batch Processing

Goal: Efficiently process multiple texts at once.

use ruvector_onnx_embeddings::{EmbedderBuilder, PretrainedModel, PoolingStrategy};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Configure for batch processing
    let mut embedder = EmbedderBuilder::new()
        .pretrained(PretrainedModel::AllMiniLmL6V2)
        .batch_size(64)           // Process 64 texts at a time
        .normalize(true)          // L2 normalize (recommended for cosine similarity)
        .pooling(PoolingStrategy::Mean)
        .build()
        .await?;

    // 2. Prepare your data
    let texts = vec![
        "Artificial intelligence is transforming technology.",
        "Machine learning models learn from data.",
        "Deep learning uses neural networks.",
        "Natural language processing understands text.",
        "Computer vision analyzes images.",
        "Reinforcement learning optimizes decisions.",
        "Vector databases enable semantic search.",
        "Embeddings capture semantic meaning.",
    ];

    // 3. Generate embeddings
    println!("Embedding {} texts...", texts.len());
    let start = std::time::Instant::now();
    let output = embedder.embed(&texts)?;
    let elapsed = start.elapsed();

    // 4. Examine results
    println!("Completed in {:?}", elapsed);
    println!("Total embeddings: {}", output.len());
    println!("Embedding dimension: {}", output.dimension);

    // 5. Show token counts per text
    println!("\nToken counts:");
    for (i, (text, tokens)) in texts.iter().zip(output.token_counts.iter()).enumerate() {
        println!("  [{}] {} tokens: \"{}...\"", i, tokens, &text[..40.min(text.len())]);
    }

    // 6. Access individual embeddings
    println!("\nFirst embedding (first 5 values):");
    let first = output.get(0).unwrap();
    println!("  [{:.4}, {:.4}, {:.4}, {:.4}, {:.4}, ...]",
        first[0], first[1], first[2], first[3], first[4]);

    Ok(())
}

Performance Table: Batch Size vs Throughput

Step 3: Building a Semantic Search Engine

Goal: Create a searchable knowledge base with semantic understanding.

use ruvector_onnx_embeddings::{
    Embedder, RuVectorBuilder, Distance
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Create embedder
    println!("Step 1: Loading embedder...");
    let embedder = Embedder::default_model().await?;

    // 2. Create search index
    println!("Step 2: Creating search index...");
    let index = RuVectorBuilder::new("programming_languages")
        .embedder(embedder)
        .distance(Distance::Cosine)      // Best for normalized embeddings
        .max_elements(100_000)           // Pre-allocate for 100k vectors
        .build()?;

    // 3. Index documents
    println!("Step 3: Indexing documents...");
    let documents = vec![
        "Rust is a systems programming language focused on safety and performance.",
        "Python is widely used for machine learning and data science applications.",
        "JavaScript is the language of the web, running in browsers everywhere.",
        "Go is designed for building scalable and efficient server applications.",
        "TypeScript adds static typing to JavaScript for better developer experience.",
        "C++ provides low-level control and high performance for system software.",
        "Java is a mature, object-oriented language popular in enterprise software.",
        "Swift is Apple's modern language for iOS and macOS development.",
        "Kotlin is a concise language that runs on the JVM, popular for Android.",
        "Haskell is a purely functional programming language with strong typing.",
    ];

    index.insert_batch(&documents)?;
    println!("   Indexed {} documents", documents.len());
    println!("   Index size: {} vectors", index.len());

    // 4. Perform searches
    println!("\nStep 4: Running searches...\n");

    let queries = vec![
        "What language is best for web development?",
        "I want to build a high-performance system application",
        "Which language should I learn for machine learning?",
        "I need a language for mobile app development",
    ];

    for query in queries {
        println!("🔍 Query: \"{}\"", query);
        let results = index.search(query, 3)?;

        for (i, result) in results.iter().enumerate() {
            println!("   {}. (score: {:.4}) {}",
                i + 1,
                result.score,
                result.text);
        }
        println!();
    }

    Ok(())
}

Search Results Table:

Step 4: Creating a RAG Pipeline

Goal: Build a retrieval-augmented generation system for LLM context.

use ruvector_onnx_embeddings::{
    Embedder, RuVectorEmbeddings, RagPipeline
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Create knowledge base
    println!("Step 1: Creating knowledge base...");
    let embedder = Embedder::default_model().await?;
    let index = RuVectorEmbeddings::new_default("ruvector_docs", embedder)?;

    // 2. Add documentation
    println!("Step 2: Adding documents...");
    let knowledge = vec![
        "RuVector is a distributed vector database that learns and adapts.",
        "RuVector uses HNSW indexing for fast approximate nearest neighbor search.",
        "The embedding dimension in RuVector is configurable based on your model.",
        "RuVector supports multiple distance metrics: Cosine, Euclidean, and Dot Product.",
        "Graph Neural Networks in RuVector improve search quality over time.",
        "RuVector integrates with ONNX models for native embedding generation.",
        "The NAPI-RS bindings allow using RuVector from Node.js applications.",
        "RuVector supports WebAssembly for running in web browsers.",
        "Quantization in RuVector reduces memory usage by up to 32x.",
        "RuVector can handle millions of vectors with sub-millisecond search.",
    ];

    index.insert_batch(&knowledge)?;

    // 3. Create RAG pipeline
    println!("Step 3: Setting up RAG pipeline...");
    let rag = RagPipeline::new(index, 3); // Retrieve top-3 documents

    // 4. Retrieve context for queries
    println!("\nStep 4: Running RAG queries...\n");

    let queries = vec![
        "How does RuVector perform search?",
        "Can I use RuVector from JavaScript?",
        "How can I reduce memory usage?",
    ];

    for query in queries {
        println!("📝 Query: \"{}\"", query);
        let context = rag.retrieve(query)?;

        println!("   Retrieved context:");
        for (i, doc) in context.iter().enumerate() {
            println!("   {}. {}", i + 1, doc);
        }

        // Format for LLM prompt
        println!("\n   LLM Prompt:");
        println!("   ───────────────────────────────────────");
        println!("   Given the following context:");
        for doc in &context {
            println!("   - {}", doc);
        }
        println!("   ");
        println!("   Answer the question: {}", query);
        println!("   ───────────────────────────────────────\n");
    }

    Ok(())
}

RAG Pipeline Flow:

┌──────────┐    ┌─────────────┐    ┌──────────┐    ┌─────────┐
│  Query   │───►│  Embedder   │───►│  Search  │───►│ Context │
│          │    │             │    │  Index   │    │         │
└──────────┘    └─────────────┘    └──────────┘    └────┬────┘
                                                        │
                                                        v
┌──────────┐    ┌─────────────┐    ┌──────────┐    ┌─────────┐
│ Response │◄───│    LLM      │◄───│  Prompt  │◄───│ Format  │
│          │    │ (external)  │    │          │    │         │
└──────────┘    └─────────────┘    └──────────┘    └─────────┘

Step 5: Text Clustering

Goal: Automatically group similar texts together.

use ruvector_onnx_embeddings::Embedder;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut embedder = Embedder::default_model().await?;

    // Mixed-category texts
    let texts = vec![
        // Technology (expected cluster 0)
        "Artificial intelligence is revolutionizing industries.",
        "Machine learning algorithms process large datasets.",
        "Neural networks mimic the human brain.",
        // Sports (expected cluster 1)
        "Football is the most popular sport worldwide.",
        "Basketball requires speed and agility.",
        "Tennis is played on different court surfaces.",
        // Food (expected cluster 2)
        "Italian pasta comes in many shapes and sizes.",
        "Sushi is a traditional Japanese dish.",
        "French cuisine is known for its elegance.",
    ];

    println!("Clustering {} texts into 3 categories...\n", texts.len());

    // Perform clustering
    let clusters = embedder.cluster(&texts, 3)?;

    // Group and display results
    let mut groups: std::collections::HashMap<usize, Vec<&str>> =
        std::collections::HashMap::new();

    for (i, &cluster) in clusters.iter().enumerate() {
        groups.entry(cluster).or_default().push(texts[i]);
    }

    println!("Clustering Results:");
    println!("═══════════════════════════════════════════");

    for (cluster_id, members) in groups.iter() {
        println!("\n📁 Cluster {}:", cluster_id);
        for text in members {
            println!("   • {}", text);
        }
    }

    Ok(())
}

Expected Clustering Output:

Configuration Reference

EmbedderConfig Options

Using EmbedderBuilder

use ruvector_onnx_embeddings::{
    EmbedderBuilder, PretrainedModel, PoolingStrategy
};

let embedder = EmbedderBuilder::new()
    .pretrained(PretrainedModel::BgeBaseEnV15)  // Choose model
    .batch_size(64)                              // Batch size
    .max_length(256)                             // Max tokens
    .pooling(PoolingStrategy::Mean)              // Pooling strategy
    .normalize(true)                             // L2 normalize
    .build()
    .await?;

Pooling Strategies

Choosing a Strategy

Text Type          Recommended Strategy
─────────────────────────────────────────
Short sentences    Mean (default)
Long documents     MeanSqrtLen
BERT fine-tuned    Cls
Keyword search     Max
Decoder models     LastToken

Performance Benchmarks

Embedding Generation Speed

Tested on AMD EPYC 7763 (64-core), Ubuntu 22.04

Memory Usage

Similarity Search Latency

API Reference

Core Types

// Main Embedder
pub struct Embedder;

impl Embedder {
    pub async fn new(config: EmbedderConfig) -> Result<Self>;
    pub async fn default_model() -> Result<Self>;
    pub async fn pretrained(model: PretrainedModel) -> Result<Self>;

    pub fn embed_one(&mut self, text: &str) -> Result<Vec<f32>>;
    pub fn embed<S: AsRef<str>>(&mut self, texts: &[S]) -> Result<EmbeddingOutput>;
    pub fn similarity(&mut self, text1: &str, text2: &str) -> Result<f32>;
    pub fn cluster<S>(&mut self, texts: &[S], n: usize) -> Result<Vec<usize>>;

    pub fn dimension(&self) -> usize;
    pub fn model_info(&self) -> &ModelInfo;
}

// Search Index
pub struct RuVectorEmbeddings;

impl RuVectorEmbeddings {
    pub fn new(name: &str, embedder: Embedder, config: IndexConfig) -> Result<Self>;
    pub fn insert(&self, text: &str, metadata: Option<Value>) -> Result<VectorId>;
    pub fn insert_batch<S>(&self, texts: &[S]) -> Result<Vec<VectorId>>;
    pub fn search(&self, query: &str, k: usize) -> Result<Vec<SearchResult>>;
    pub fn len(&self) -> usize;
}

// RAG Pipeline
pub struct RagPipeline;

impl RagPipeline {
    pub fn new(index: RuVectorEmbeddings, top_k: usize) -> Self;
    pub fn retrieve(&self, query: &str) -> Result<Vec<String>>;
    pub fn add_documents<S>(&mut self, docs: &[S]) -> Result<Vec<VectorId>>;
}

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                     RuVector ONNX Embeddings                             │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌───────────┐ │
│  │    Text     │ -> │  Tokenizer  │ -> │    ONNX     │ -> │  Pooling  │ │
│  │   Input     │    │ (HF Rust)   │    │   Runtime   │    │  Strategy │ │
│  └─────────────┘    └─────────────┘    └─────────────┘    └───────────┘ │
│                                                                  │       │
│                                                                  v       │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌───────────┐ │
│  │   Search    │ <- │   Vector    │ <- │  Normalize  │ <- │ Embedding │ │
│  │  Results    │    │    Index    │    │   (L2)      │    │  Vector   │ │
│  └─────────────┘    └─────────────┘    └─────────────┘    └───────────┘ │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Troubleshooting

Common Issues and Solutions

Debugging Tips

// Enable verbose logging
std::env::set_var("RUST_LOG", "debug");
tracing_subscriber::fmt::init();

// Check model loading
let embedder = Embedder::default_model().await?;
println!("Model: {}", embedder.model_info().name);
println!("Dimension: {}", embedder.dimension());

Running Benchmarks

# Run all benchmarks
cargo bench

# Generate HTML report
cargo bench -- --verbose
open target/criterion/report/index.html

Examples

# Basic embedding
cargo run --example basic_embedding

# Batch processing
cargo run --example batch_embedding

# Semantic search
cargo run --example semantic_search

# Full interactive demo
cargo run

License

MIT License - See LICENSE for details.

Built with Rust for the RuVector ecosystem.