LumosAI Vector Core

Core abstractions and traits for the LumosAI vector storage system. This crate provides the foundational types, traits, and utilities that all vector database implementations in the LumosAI ecosystem build upon.

Features

• Unified Vector Storage Interface: Common traits for all vector database implementations
• Type-Safe Operations: Strongly typed vector operations with compile-time guarantees
• Async-First Design: Built for high-performance async applications
• Flexible Configuration: Extensible configuration system for different vector databases
• Performance Monitoring: Built-in performance metrics and monitoring capabilities
• Error Handling: Comprehensive error types for robust error handling

Core Traits

VectorStorage

The main trait that all vector database implementations must implement:

use lumosai_vector_core::{VectorStorage, VectorData, SearchRequest};

#[async_trait]
pub trait VectorStorage: Send + Sync {
    async fn insert(&self, vectors: Vec<VectorData>) -> Result<Vec<String>>;
    async fn search(&self, request: SearchRequest) -> Result<SearchResponse>;
    async fn delete(&self, ids: Vec<String>) -> Result<()>;
    async fn get(&self, ids: Vec<String>) -> Result<Vec<Option<VectorData>>>;
}

EmbeddingProvider

For embedding generation:

use lumosai_vector_core::EmbeddingProvider;

#[async_trait]
pub trait EmbeddingProvider: Send + Sync {
    async fn embed_texts(&self, texts: Vec<String>) -> Result<Vec<Vec<f32>>>;
    async fn embed_query(&self, query: String) -> Result<Vec<f32>>;
}

Types

VectorData

Represents a vector with its metadata:

use lumosai_vector_core::VectorData;

let vector = VectorData {
    id: "doc_1".to_string(),
    vector: vec![0.1, 0.2, 0.3, 0.4],
    metadata: Some(json!({
        "title": "Document Title",
        "content": "Document content..."
    })),
};

SearchRequest

Configuration for vector similarity search:

use lumosai_vector_core::{SearchRequest, SimilarityMetric};

let request = SearchRequest {
    vector: vec![0.1, 0.2, 0.3, 0.4],
    limit: 10,
    similarity_threshold: Some(0.8),
    metric: SimilarityMetric::Cosine,
    filter: None,
};

Configuration

The crate provides flexible configuration options:

use lumosai_vector_core::{VectorConfig, SimilarityMetric};

let config = VectorConfig {
    dimension: 384,
    metric: SimilarityMetric::Cosine,
    index_type: "HNSW".to_string(),
    parameters: json!({
        "ef_construction": 200,
        "m": 16
    }),
};

Error Handling

Comprehensive error types for robust error handling:

use lumosai_vector_core::VectorError;

match storage.insert(vectors).await {
    Ok(ids) => println!("Inserted vectors with IDs: {:?}", ids),
    Err(VectorError::ConnectionError(msg)) => {
        eprintln!("Connection failed: {}", msg);
    }
    Err(VectorError::ValidationError(msg)) => {
        eprintln!("Validation failed: {}", msg);
    }
    Err(err) => eprintln!("Other error: {}", err),
}

Performance Monitoring

Built-in performance metrics:

use lumosai_vector_core::PerformanceMetrics;

let metrics = storage.get_metrics().await?;
println!("Average search latency: {}ms", metrics.avg_search_latency_ms);
println!("Total operations: {}", metrics.total_operations);

Usage with Vector Databases

This crate is designed to be used with specific vector database implementations:

• lumosai-vector-lancedb - LanceDB implementation
• lumosai-vector-qdrant - Qdrant implementation
• lumosai-vector-weaviate - Weaviate implementation
• lumosai-vector-milvus - Milvus implementation
• lumosai-vector-postgres - PostgreSQL with pgvector

Example

use lumosai_vector_core::{VectorStorage, VectorData, SearchRequest};

async fn example_usage<T: VectorStorage>(storage: T) -> Result<(), Box<dyn std::error::Error>> {
    // Insert vectors
    let vectors = vec![
        VectorData {
            id: "1".to_string(),
            vector: vec![0.1, 0.2, 0.3],
            metadata: None,
        }
    ];
    
    let ids = storage.insert(vectors).await?;
    println!("Inserted: {:?}", ids);
    
    // Search for similar vectors
    let request = SearchRequest {
        vector: vec![0.1, 0.2, 0.3],
        limit: 5,
        similarity_threshold: Some(0.8),
        ..Default::default()
    };
    
    let results = storage.search(request).await?;
    println!("Found {} similar vectors", results.results.len());
    
    Ok(())
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.