Lumosai Vector Storage System

A unified, high-performance vector storage system for Lumos.ai that provides a consistent interface across multiple storage backends.

🚀 Features

• Unified Interface: Single API for all vector storage backends
• Multiple Backends: Memory, Qdrant, PostgreSQL, MongoDB, and more
• High Performance: Optimized for speed and scalability
• Type Safety: Strong typing with comprehensive error handling
• Async/Await: Full async support with tokio
• Extensible: Easy to add new storage backends

📁 Architecture

lumosai_vector/
├── core/                      # Core abstractions and traits
├── memory/                    # In-memory storage implementation
├── qdrant/                    # Qdrant vector database integration
├── postgres/                  # PostgreSQL with pgvector support
├── mongodb/                   # MongoDB vector search (coming soon)
└── vectorize/                 # Cloudflare Vectorize (coming soon)

🔧 Quick Start

Add to your Cargo.toml:

[dependencies]
lumosai_vector = { path = "../lumosai_vector", features = ["memory"] }

Basic Usage

use lumosai_vector::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a memory storage instance
    let storage = lumosai_vector::memory::MemoryVectorStorage::new().await?;
    
    // Create an index
    let config = IndexConfig::new("documents", 384)
        .with_metric(SimilarityMetric::Cosine);
    storage.create_index(config).await?;
    
    // Insert documents
    let docs = vec![
        Document::new("doc1", "Hello world")
            .with_embedding(vec![0.1; 384])
            .with_metadata("type", "greeting"),
    ];
    storage.upsert_documents("documents", docs).await?;
    
    // Search
    let request = SearchRequest::new("documents", vec![0.1; 384])
        .with_top_k(5);
    let results = storage.search(request).await?;
    
    println!("Found {} results", results.results.len());
    Ok(())
}

🗄️ Storage Backends

Memory Storage

Fast in-memory storage for development and testing:

use lumosai_vector::memory::MemoryVectorStorage;

let storage = MemoryVectorStorage::new().await?;

Features:

• ✅ Fast in-memory operations
• ✅ Perfect for development and testing
• ✅ No external dependencies
• ❌ Data is not persistent

Qdrant Storage

High-performance vector database (requires qdrant feature):

lumosai_vector = { features = ["qdrant"] }

use lumosai_vector::qdrant::QdrantVectorStorage;

let storage = QdrantVectorStorage::new("http://localhost:6334").await?;

Features:

• ✅ High-performance vector search
• ✅ Distributed and scalable
• ✅ Advanced filtering capabilities
• ✅ Production-ready

PostgreSQL Storage

SQL database with pgvector extension (requires postgres feature):

lumosai_vector = { features = ["postgres"] }

use lumosai_vector::postgres::PostgresVectorStorage;

let storage = PostgresVectorStorage::new("postgresql://user:pass@localhost/db").await?;

Features:

• ✅ ACID transactions
• ✅ SQL integration
• ✅ Rich metadata queries
• ✅ Mature ecosystem

🔧 Configuration

Features

• default = ["memory"] - Includes memory storage
• memory - In-memory storage implementation
• qdrant - Qdrant vector database support
• postgres - PostgreSQL with pgvector support
• all - All available backends

Auto-Detection

The system can automatically detect and use the best available backend:

use lumosai_vector::utils;

let storage = utils::create_auto_storage().await?;

This will try backends in order of preference:

1. Qdrant (if available and configured)
2. PostgreSQL (if DATABASE_URL is set)
3. Memory (fallback)

🧪 Testing

Run tests for all modules:

cd lumosai_vector
cargo test --workspace

Run tests for specific backend:

cargo test -p lumosai-vector-memory
cargo test -p lumosai-vector-qdrant --features qdrant

🔄 Migration from Old Architecture

This unified module replaces the previous separate crates:

• lumos-vector-core → lumosai_vector::core
• lumos-vector-memory → lumosai_vector::memory
• lumos-vector-qdrant → lumosai_vector::qdrant

The API remains the same, only import paths have changed.

📈 Performance

Benchmark results on a typical development machine:

🤝 Contributing

1. Add new storage backends in their own subdirectory
2. Implement the VectorStorage trait
3. Add feature flags and documentation
4. Include comprehensive tests

📄 License

MIT OR Apache-2.0