OSIS - Object Storage with SmartID

OSIS provides filesystem-based object storage with distributed, human-readable identifiers.

Features

• SmartID (SID): Short, collision-free IDs (base-36, 4-6 chars)
• Filesystem Storage: Objects stored as OTOML files (compact TOML format)
• OSchema: Define object schemas in .oschema files
• Auto-generated Code: include_oschema! generates structs, builders, storage API, and Rhai bindings
• Full-text Search: Tantivy-powered search on indexed fields
• RPC Server: Access storage via HTTP, Unix socket, or Redis queues
• Rhai Integration: Auto-generated namespaced functions for scripting

Integrated Modules

This package now includes both OTOML and OSchema modules:

OTOML - Canonical Serialization

Deterministic TOML serialization with native support for four canonical types:

• otoml::OTime - UTC timestamp (4 bytes)
• otoml::OCur - Currency/asset amounts (integer micro-units)
• otoml::OLocation - Geographic coordinates with uncertainty radius
• otoml::OAddress - Planet-scale civic addresses

Functions:

• dump_otoml() / load_otoml() - Text serialization
• dump_obin() / load_obin() - Binary serialization

OSchema - Schema Definition Language

Minimal schema language for defining structured data with OTOML type support:

• oschema::parse_schema() - Parse schema definitions
• oschema::to_json_schema() - Convert to JSON Schema
• oschema::to_rust_structs() - Generate Rust code
• oschema::to_markdown() - Generate documentation
• oschema::to_html() - Generate interactive HTML docs

Table of Contents

• Quick Start
• OSchema
• Rust Usage
• Rhai Integration
• RPC Server
• Full-text Search
• SmartID
• Running Examples
• Adding Custom Methods
• Exposing Custom Methods via RPC
• API Reference

Quick Start

1. Define your schema (task.oschema)

Status = "pending" | "in_progress" | "done"
Priority = "low" | "medium" | "high"

Task = {
    sid: str              # SmartID (required for OSIS)
    name?: str            # optional, used in filename [index]
    title: str            # task title [index]
    description?: str     # optional description [index]
    status: Status
    priority: Priority
    created: time
    tags: [str]
}

2. Use in Rust

use herolib_derive::include_oschema;

// Generate: Task struct, Status/Priority enums, TaskOSIS storage wrapper
include_oschema!("task.oschema");

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create storage
    let mut storage = TaskOSIS::new("~/data", "myproject", 1)?;

    // Create and store a task
    let task = storage.create()?
        .with_title("Implement auth")
        .with_priority(Priority::High);
    storage.set(&task)?;

    // Load, search, delete
    let loaded = storage.get(&task.sid)?;
    let found = storage.find("auth")?;
    storage.delete(&task.sid)?;

    Ok(())
}

3. Use in Rhai

// Functions are auto-generated: osis_{namespace}_{type}_{operation}()
let task = osis_myproject_task_new();
task.title = "Implement auth";
task.priority = "high";
osis_myproject_task_set(task);

let loaded = osis_myproject_task_get(task.sid);
osis_myproject_task_delete(task.sid);

OSchema

Define object schemas in .oschema files:

# Enumerations
Status = "pending" | "in_progress" | "done"

# Struct with fields
Task = {
    sid: str              # SmartID (required for OSIS storage)
    name?: str            # optional field (note the ?)
    title: str            # required string field [index]
    count: u64            # integer field
    active: bool          # boolean field
    created: time         # timestamp (OTime)
    tags: [str]           # list of strings
    meta: {str:any}       # map
    status: Status        # reference to enum
}

Field Markers

• ? after field name = optional (wrapped in Option<T>)
• [index] in comment = enable full-text search on this field

Supported Types

OSchema	Rust
str	String
bool	bool
u8, u16, u32, u64	u8, u16, u32, u64
i8, i16, i32, i64	i8, i16, i32, i64
f32, f64	f32, f64
time	OTime
[T]	Vec<T>
{K:V}	HashMap<K, V>
any	serde_json::Value

Rust Usage

Using include_oschema!

The macro generates everything at compile time:

use herolib_derive::include_oschema;
use herolib_osis::OTime;
use serde::{Serialize, Deserialize};

// Generate types from schema
include_oschema!("schemas/task.oschema");

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // TaskOSIS is auto-generated with CRUD methods
    let mut storage = TaskOSIS::new("~/hero/var/osis", "myproject", 1)?;

    // create() generates a new SID
    let mut task = storage.create()?;
    task.title = "My Task".to_string();
    task.status = Status::Pending;
    
    // Or use builder pattern
    let task = storage.create()?
        .with_title("My Task")
        .with_status(Status::Pending)
        .with_priority(Priority::High);

    // CRUD operations
    storage.set(&task)?;                    // Store
    let loaded = storage.get(&task.sid)?;   // Load
    let exists = storage.exists(&task.sid)?; // Check
    storage.delete(&task.sid)?;             // Delete

    // List and search
    let all_sids = storage.list(None)?;
    let filtered = storage.list(Some("auth"))?;  // Filter by name prefix
    let found = storage.find("title:auth*")?;    // Full-text search

    Ok(())
}

Generated API

For a type Task, include_oschema! generates:

Generated	Description
Task struct	With all fields, Default, Serialize, Deserialize
Task::with_*()	Builder methods for each field
TaskOSIS struct	Storage wrapper with CRUD methods
TaskOSIS::new()	Create storage instance
TaskOSIS::create()	Create object with new SID
TaskOSIS::set(&obj)	Store object
TaskOSIS::get(sid)	Load by SID
TaskOSIS::delete(sid)	Delete by SID
TaskOSIS::exists(sid)	Check existence
TaskOSIS::list(filter)	List all SIDs
TaskOSIS::find(query)	Full-text search
OsisRpcHandler impl	For RPC server integration
register_rhai_*()	For Rhai integration

File Storage Format

Objects are stored as OTOML files:

{base_path}/{collection}/{type}/{sid}.otoml           # without name
{base_path}/{collection}/{type}/{name}_{sid}.otoml    # with name

Example: ~/hero/var/osis/myproject/task/auth_0001.otoml

Rhai Integration

OSIS auto-generates namespaced Rhai functions for each type.

Enabling Rhai

[dependencies]
herolib-osis = { version = "0.1", features = ["rhai"] }

Registering Functions

In your Rust code, register the Rhai functions:

use herolib_derive::include_oschema;
use rhai::Engine;

include_oschema!("task.oschema");
include_oschema!("user.oschema");

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut engine = Engine::new();

    // Register LOCAL storage (filesystem + Tantivy index)
    // Functions: osis_myapp_task_new(), osis_myapp_task_get(), etc.
    TaskOSIS::register_rhai_local(&mut engine, "myapp", "/tmp/data", 1)?;
    UserOSIS::register_rhai_local(&mut engine, "myapp", "/tmp/data", 1)?;

    // Register RPC clients (requires "rpc" feature)
    #[cfg(feature = "rpc")]
    {
        // HTTP: osis_http_task_new(), osis_http_task_get(), etc.
        TaskOSIS::register_rhai_http(&mut engine, "http", "http://localhost:7352/osis")?;

        // Unix Socket: osis_sock_task_new(), etc.
        TaskOSIS::register_rhai_socket(&mut engine, "sock", "/tmp/osis.sock")?;

        // Redis Queue: osis_redis_task_new(), etc.
        TaskOSIS::register_rhai_redis(&mut engine, "redis", "redis://localhost:6379", "demo")?;
    }

    // Run Rhai script
    engine.run(r#"
        let task = osis_myapp_task_new();
        task.title = "From Rhai";
        osis_myapp_task_set(task);
    "#)?;

    Ok(())
}

Generated Rhai Functions

For namespace myapp and type task:

Function	Description
osis_myapp_task_new()	Create new object with SID
osis_myapp_task_get(sid)	Get object by SID
osis_myapp_task_set(obj)	Store object
osis_myapp_task_delete(sid)	Delete object
osis_myapp_task_exists(sid)	Check if exists
osis_myapp_task_list()	List all SIDs
osis_myapp_task_find(query)	Search objects

Rhai Script Example

// Create and store a task
let task = osis_myapp_task_new();
task.title = "Implement feature";
task.name = "feature_x";
task.priority = "high";
task.status = "pending";
task.tags = ["backend", "api"];
osis_myapp_task_set(task);
print("Created task: " + task.sid);

// Load it back
let loaded = osis_myapp_task_get(task.sid);
print("Loaded: " + loaded.title);

// Update
loaded.status = "in_progress";
osis_myapp_task_set(loaded);

// List all
let all = osis_myapp_task_list();
print("Total tasks: " + all.len());

// Search
let found = osis_myapp_task_find("feature");
print("Found: " + found.len());

// Delete
osis_myapp_task_delete(task.sid);

4 Access Modes from Rhai

1. Local Storage - Direct filesystem access with Tantivy indexing
2. HTTP RPC - Access remote server via HTTP REST API
3. Socket RPC - Access remote server via Unix socket
4. Redis RPC - Access remote server via Redis queues

// Same API, different backends (configured in Rust)
let local_task = osis_local_task_new();    // Local filesystem
let http_task = osis_http_task_new();      // Via HTTP
let sock_task = osis_sock_task_new();      // Via Unix socket
let redis_task = osis_redis_task_new();    // Via Redis queue

RPC Server

OSIS includes an RPC server for remote access via HTTP, Unix socket, or Redis queues.

Enabling RPC

[dependencies]
herolib-osis = { version = "0.1", features = ["rpc"] }

Starting the Server

use herolib_derive::include_oschema;
use herolib_osis::rpc::{OsisRpcConfig, OsisRpcServer};

include_oschema!("task.oschema");
include_oschema!("user.oschema");

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize tracing for logging
    tracing_subscriber::fmt()
        .with_env_filter(
            tracing_subscriber::EnvFilter::from_default_env()
                .add_directive("herolib_osis=info".parse()?)
        )
        .init();

    // Create storage instances
    let storage_task = TaskOSIS::new("~/data", "demo", 1)?;
    let storage_user = UserOSIS::new("~/data", "demo", 1)?;

    // Configure server
    let config = OsisRpcConfig::new("redis://127.0.0.1:6379")
        .context("demo")                    // Redis key namespace
        .socket("/tmp/osis.sock")           // Unix socket
        .http("127.0.0.1:7352")             // HTTP server
        .http_prefix("/osis");              // HTTP path prefix

    // Create and start server
    let server = OsisRpcServer::new(config).await?;
    server.register("task", storage_task).await;
    server.register("user", storage_user).await;

    println!("Server running on:");
    println!("  HTTP:   http://127.0.0.1:7352/osis/");
    println!("  Socket: /tmp/osis.sock");
    println!("  Redis:  redis://127.0.0.1:6379 (context: demo)");

    server.run().await?;
    Ok(())
}

Server Logging

The RPC server uses tracing for structured logging. Configure the log level via the RUST_LOG environment variable:

# Info level - shows HTTP requests, RPC methods, and completion status
RUST_LOG=herolib_osis=info cargo run --example task_rpc --features rpc

# Debug level - includes request data, params, and job IDs
RUST_LOG=herolib_osis=debug cargo run --example task_rpc --features rpc

# Trace level - maximum verbosity
RUST_LOG=herolib_osis=trace cargo run --example task_rpc --features rpc

Example log output:

INFO  herolib_osis::rpc::server > Registering RPC handler for type: task
INFO  herolib_osis::rpc::server > HTTP server listening on: http://127.0.0.1:7352/osis/
INFO  herolib_osis::rpc::server > Starting Redis queue processor
INFO  herolib_osis::rpc::server > HTTP POST /osis/task/rpc
INFO  herolib_osis::rpc::server > JSON-RPC: task.set (id=1)
INFO  herolib_osis::rpc::server > Processing task.Set job_id=jsonrpc_123456789
INFO  herolib_osis::rpc::server > Completed task.Set job_id=jsonrpc_123456789 status=Ok

HTTP REST API

Method	Path	Description
GET	/osis/{type}	List all SIDs
GET	/osis/{type}/{sid}	Get object by SID
POST	/osis/{type}	Create/update object
POST	/osis/{type}/new	Get template with new SID
POST	/osis/{type}/find	Search (body: {"query": "..."})
DELETE	/osis/{type}/{sid}	Delete object

HTTP Examples

# Get new template with SID
curl -X POST http://localhost:7352/osis/task/new

# Create task
curl -X POST http://localhost:7352/osis/task \
  -H "Content-Type: application/json" \
  -d '{"sid":"0001","title":"My Task","status":"pending","priority":"high"}'

# Get task
curl http://localhost:7352/osis/task/0001

# List all tasks
curl http://localhost:7352/osis/task

# Search
curl -X POST http://localhost:7352/osis/task/find \
  -H "Content-Type: application/json" \
  -d '{"query":"title:My*"}'

# Delete
curl -X DELETE http://localhost:7352/osis/task/0001

Unix Socket Protocol

Line-based protocol:

{type}
{method}
{job_id}
{data...}

# Get task
echo -e "task\nget\njob123\n0001" | nc -U /tmp/osis.sock

# List tasks
echo -e "task\nlist\njob124" | nc -U /tmp/osis.sock

Redis Queue Protocol

Requests are pushed to osis:{context}:queue:{type}, results stored in osis:{context}:result:{job_id}.

# Push request
redis-cli LPUSH osis:demo:queue:task "get\njob456\n0001"

# Get result
redis-cli GET osis:demo:result:job456

Full-text Search

OSIS uses Tantivy for full-text search on indexed fields.

Marking Fields for Indexing

Add [index] to field comments:

Task = {
    sid: str
    title: str        # [index]
    description?: str # [index]
    status: Status    # not indexed
}

Search Syntax

// Simple search (all indexed fields)
storage.find("authentication")?;

// Field-specific
storage.find("title:auth")?;

// Prefix/wildcard
storage.find("auth*")?;
storage.find("title:impl*")?;

// Boolean operators
storage.find("auth AND api")?;
storage.find("title:auth OR description:jwt")?;
storage.find("cloud NOT deprecated")?;

// Phrase search
storage.find("\"user authentication\"")?;

// Fuzzy search
storage.find("clud~1")?;  // matches "cloud"

Index Location

{base_path}/{collection}/{type}/.index/

SmartID

SmartID provides short, collision-free identifiers for distributed systems.

Format

• Base-36 alphabet: 0123456789abcdefghijklmnopqrstuvwxyz
• 4-6 characters
• Supports up to 999 contributors

Capacity

Length	Total IDs	Per contributor
4	1,679,616	~1,681
5	60,466,176	~60,526
6	2,176,782,336	~2,178,960

Direct Usage

use herolib_osis::sid::{Space, SmartId};

let mut space = Space::new("myproject");
space.register_contributor()?;

let sid = space.mint(0)?;  // "0000"
let parsed = SmartId::parse("abc1")?;

Running Examples

Run Tests

cd packages/osis

# All tests
cargo test --features "rpc rhai"

# Specific test
cargo test --features rpc test_http

Run RPC Server

# Start server (requires Redis running)
cargo run --example task_rpc --features rpc

Run Rhai Engine Test

# Test local storage with Rhai
cargo run --example rhai_engine_test --features "rhai rpc"

Test HTTP API

# Start server first, then:
curl http://localhost:7352/osis/task
curl -X POST http://localhost:7352/osis/task/new

Adding Custom Methods

Since include_oschema! generates structs at compile time, you can extend them with your own methods by adding additional impl blocks. Rust allows multiple impl blocks for the same struct.

Adding Methods to the Data Struct

use herolib_derive::include_oschema;
use herolib_osis::OTime;
use serde::{Deserialize, Serialize};

// Generate Task, Status, Priority, TaskOSIS from schema
include_oschema!("task.oschema");

// Add custom methods to Task
impl Task {
    /// Check if the task is overdue.
    pub fn is_overdue(&self) -> bool {
        if let Some(due) = &self.due {
            due < &OTime::now()
        } else {
            false
        }
    }

    /// Mark the task as completed.
    pub fn complete(&mut self) {
        self.status = Status::Done;
    }

    /// Check if task is high priority.
    pub fn is_urgent(&self) -> bool {
        matches!(self.priority, Priority::High | Priority::Critical)
    }

    /// Get a formatted display string.
    pub fn display(&self) -> String {
        format!("[{:?}] {} ({})", self.priority, self.title, self.sid)
    }
}

Adding Methods to the Storage Wrapper

// Add custom methods to TaskOSIS
impl TaskOSIS {
    /// Get all high-priority tasks.
    pub fn get_urgent_tasks(&self) -> Result<Vec<Task>, String> {
        let all_sids = self.list(None)?;
        let mut urgent = Vec::new();
        for sid in all_sids {
            let task = self.get(&sid)?;
            if task.is_urgent() {
                urgent.push(task);
            }
        }
        Ok(urgent)
    }

    /// Get all overdue tasks.
    pub fn get_overdue_tasks(&self) -> Result<Vec<Task>, String> {
        let all_sids = self.list(None)?;
        let mut overdue = Vec::new();
        for sid in all_sids {
            let task = self.get(&sid)?;
            if task.is_overdue() {
                overdue.push(task);
            }
        }
        Ok(overdue)
    }

    /// Complete a task by SID.
    pub fn complete_task(&self, sid: &str) -> Result<(), String> {
        let mut task = self.get(sid)?;
        task.complete();
        self.set(&task)
    }

    /// Get tasks by status.
    pub fn get_by_status(&self, status: Status) -> Result<Vec<Task>, String> {
        let all_sids = self.list(None)?;
        let mut result = Vec::new();
        for sid in all_sids {
            let task = self.get(&sid)?;
            if task.status == status {
                result.push(task);
            }
        }
        Ok(result)
    }
}

Using Custom Methods

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut storage = TaskOSIS::new("~/data", "myproject", 1)?;

    // Create a task
    let task = storage.create()?
        .with_title("Important feature")
        .with_priority(Priority::Critical);
    storage.set(&task)?;

    // Use custom Task methods
    println!("{}", task.display());
    println!("Urgent: {}", task.is_urgent());

    // Use custom TaskOSIS methods
    let urgent = storage.get_urgent_tasks()?;
    println!("Found {} urgent tasks", urgent.len());

    // Complete via storage method
    storage.complete_task(&task.sid)?;

    // Get by status
    let done = storage.get_by_status(Status::Done)?;
    println!("Completed tasks: {}", done.len());

    Ok(())
}

See examples/basic/task_custom.rs for a complete working example.

Exposing Custom Methods via RPC

You can expose custom methods via RPC using the osis_rpc_impl! macro and #[osis_rpc] attribute.

Defining Custom RPC Methods

use herolib_derive::{include_oschema, osis_rpc, osis_rpc_impl};

include_oschema!("task.oschema");

// Wrap your impl block with osis_rpc_impl! and mark methods with #[osis_rpc]
osis_rpc_impl!(TaskOSIS {
    /// Get all high-priority tasks.
    #[osis_rpc]
    pub fn get_urgent_tasks(&self) -> Result<Vec<Task>, String> {
        let all_sids = self.list(None)?;
        let mut urgent = Vec::new();
        for sid in all_sids {
            let task = self.get(&sid)?;
            if matches!(task.priority, Priority::High | Priority::Critical) {
                urgent.push(task);
            }
        }
        Ok(urgent)
    }

    /// Complete a task by SID.
    #[osis_rpc]
    pub fn complete_task(&self, sid: &str) -> Result<Task, String> {
        let mut task = self.get(sid)?;
        task.status = Status::Done;
        self.set(&task)?;
        Ok(task)
    }

    /// Get tasks by status.
    #[osis_rpc]
    pub fn get_by_status(&self, status: String) -> Result<Vec<Task>, String> {
        // implementation
    }
});

Calling Custom Methods via HTTP

# Call method with no parameters
curl -X POST http://localhost:7352/osis/task/call/get_urgent_tasks

# Call method with string parameter
curl -X POST http://localhost:7352/osis/task/call/complete_task \
  -H "Content-Type: application/json" \
  -d '"abc123"'

# Call method with typed parameter
curl -X POST http://localhost:7352/osis/task/call/get_by_status \
  -H "Content-Type: application/json" \
  -d '"pending"'

Calling Custom Methods via Line Protocol

# Format: call:method_name
echo -e "call:get_urgent_tasks\njob123\n{}" | nc -U /tmp/osis.sock

# With parameter
echo -e "call:complete_task\njob124\n\"abc123\"" | nc -U /tmp/osis.sock

Custom RPC Method Rules

1. Methods must take &self (not &mut self)
2. Methods must return Result<T, String> where T: Serialize
3. Parameters must implement Deserialize
4. String parameters (&str) are passed directly from request data
5. Other parameters are parsed as JSON

See examples/basic/task_custom_rpc.rs for a complete RPC server example.

API Reference

DBTyped (Type-safe Storage)

use herolib_osis::db::{DBTyped, OsisObject};
use herolib_osis::sid::SmartId;

// Create database without indexing
let db: DBTyped<Task> = DBTyped::new(data_dir, user_id)?;

// Create database with full-text indexing (for types with @index fields)
let db: DBTyped<Task> = DBTyped::new_with_index(data_dir, index_dir, user_id)?;

// CRUD operations
let mut task = Task::default();
task.title = "My Task".to_string();
db.set(&mut task)?;                      // Store (generates SID if empty)
let loaded = db.get(&task.sid)?;         // Load by SmartId
db.delete(&task.sid)?;                   // Delete
let exists = db.exists(&task.sid)?;      // Check existence
let sids = db.list()?;                   // List all SmartIds

// Search (requires new_with_index)
let found = db.search("query")?;         // Full-text search
db.rebuild_index()?;                     // Rebuild search index

OsisObject Trait

Types stored in DBTyped must implement OsisObject:

use herolib_derive::OsisObject;
use herolib_osis::sid::SmartId;

#[derive(Default, Serialize, Deserialize, OsisObject)]
#[osis(index = "title, description")]  // Fields for full-text search
pub struct Task {
    pub sid: SmartId,
    pub title: String,
    pub description: Option<String>,
    pub status: Status,
}

Generated Domain Handler (OsisApp)

// Generated from OSchema
let app = OsisApp::new(db_path, index_path, user_id)?;

// CRUD per type
let task = app.task_new();               // Create with empty SID
app.task_set(&mut task)?;                // Store (generates SID)
let loaded = app.task_get(&sid)?;        // Load by SID string
app.task_delete(&sid)?;                  // Delete
let exists = app.task_exists(&sid);      // Check existence
let sids = app.task_list();              // List all SIDs
let found = app.task_find("query")?;     // Full-text search
app.task_rebuild_index()?;               // Rebuild index

OsisRpcConfig

let config = OsisRpcConfig::new("redis://127.0.0.1:6379")
    .context("mycontext")           // Redis key namespace
    .socket("/tmp/osis.sock")       // Unix socket path
    .http("127.0.0.1:7352")         // HTTP bind address
    .http_prefix("/osis");          // HTTP path prefix

Rhai Registration

// Local storage
TaskOSIS::register_rhai_local(&mut engine, namespace, base_path, user_id)?;

// RPC clients (requires "rpc" feature)
TaskOSIS::register_rhai_http(&mut engine, namespace, base_url)?;
TaskOSIS::register_rhai_socket(&mut engine, namespace, socket_path)?;
TaskOSIS::register_rhai_redis(&mut engine, namespace, redis_url, context)?;

License

Apache-2.0