HeroIndex

A high-performance full-text search server built on Tantivy, exposing an OpenRPC interface over Unix sockets.

Repository: https://forge.ourworld.tf/lhumina_research/hero_index_server

Looking for the client library? See heroindex_client for easy integration into your Rust applications.

Features

• Multiple Index Management - Create, delete, and manage multiple search indexes
• Dynamic Schemas - Define custom schemas with 10+ field types
• Powerful Queries - Full-text, fuzzy, phrase, boolean, range, regex queries
• OpenRPC Discovery - Self-documenting API via rpc.discover
• Concurrent Connections - Handle multiple clients simultaneously
• Fast Fields - Columnar storage for sorting and aggregations
• Zero-Copy Search - Efficient memory-mapped index files

Installation

From crates.io

cargo install heroindex

From source

git clone https://forge.ourworld.tf/lhumina_research/hero_index_server.git
cd hero_index_server
cargo build --release

Quick Start

1. Start the Server

heroindex --dir /var/lib/heroindex --socket /tmp/heroindex.sock

2. Connect with the Client Library

Use heroindex_client to connect:

use heroindex_client::HeroIndexClient;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), heroindex_client::Error> {
    let mut client = HeroIndexClient::connect("/tmp/heroindex.sock").await?;
    
    // Create an index
    client.db_create("articles", json!({
        "fields": [
            {"name": "title", "type": "text", "stored": true, "indexed": true},
            {"name": "body", "type": "text", "stored": true, "indexed": true}
        ]
    })).await?;
    
    // Add documents
    client.db_select("articles").await?;
    client.doc_add(json!({"title": "Hello", "body": "World"})).await?;
    client.commit().await?;
    client.reload().await?;
    
    // Search
    let results = client.search(
        json!({"type": "match", "field": "body", "value": "world"}),
        10, 0
    ).await?;
    
    println!("Found {} results", results.total_hits);
    Ok(())
}

Command Line Options

heroindex [OPTIONS]

Options:
  -d, --dir <DIR>        Base directory for all indexes
  -s, --socket <SOCKET>  Unix socket path for RPC interface
  -h, --help             Print help
  -V, --version          Print version

Schema Definition

Define your index schema with these field types:

Example Schema

{
  "fields": [
    {"name": "id", "type": "str", "stored": true, "indexed": true},
    {"name": "title", "type": "text", "stored": true, "indexed": true, "tokenizer": "en_stem"},
    {"name": "content", "type": "text", "stored": true, "indexed": true},
    {"name": "views", "type": "u64", "stored": true, "indexed": true, "fast": true},
    {"name": "rating", "type": "f64", "stored": true, "indexed": true, "fast": true},
    {"name": "published", "type": "date", "stored": true, "indexed": true, "fast": true},
    {"name": "active", "type": "bool", "stored": true, "indexed": true},
    {"name": "metadata", "type": "json", "stored": true, "indexed": true}
  ]
}

Query Types

Match Query (Full-Text)

{"type": "match", "field": "content", "value": "search terms"}

Term Query (Exact)

{"type": "term", "field": "id", "value": "abc123"}

Fuzzy Query (Typo-Tolerant)

{"type": "fuzzy", "field": "title", "value": "serch", "distance": 2}

Phrase Query

{"type": "phrase", "field": "content", "value": "exact phrase match"}

Prefix Query

{"type": "prefix", "field": "title", "value": "hel"}

Range Query

{"type": "range", "field": "views", "gte": 100, "lt": 1000}

Regex Query

{"type": "regex", "field": "title", "value": "test.*"}

Boolean Query

{
  "type": "boolean",
  "must": [{"type": "match", "field": "content", "value": "rust"}],
  "should": [{"type": "match", "field": "title", "value": "tutorial"}],
  "must_not": [{"type": "term", "field": "status", "value": "draft"}]
}

RPC Methods

Performance Tips

1. Use batch inserts - doc.add_batch is much faster than individual adds
2. Commit periodically - Don't commit after every document
3. Enable fast fields - For fields used in sorting/filtering
4. Use appropriate tokenizers - en_stem for English, raw for keywords

Related Crates

• heroindex_client - Client library for connecting to HeroIndex

License

MIT License - see LICENSE for details.

Credits

Built on the excellent Tantivy search engine library.