Image Classifier Node

Hardware-agnostic image classification using ONNX models from the model catalog.

Features

• On-Demand Classification: Triggered by UI, behavior trees, or topics (no wasted CPU!)
• Hardware Agnostic: Works with simulation, real cameras, or fake drivers
• Model Catalog Integration: Auto-loads models with preprocessing metadata
• Configurable Processing Rate: Throttle FPS when in continuous mode (default: 10 Hz)
• Top-K Predictions: Returns top 5 most confident classes
• ImageNet Support: 1000-class classification (MobileNet, ResNet, etc.)

Topic Interface

Control: /vision/classification/control (ClassificationControl - triggers) Input: /robot/sensors/camera/rgb (CameraImage - processed when enabled) Output: /vision/classification (Classification - results)

Hardware Sources (all compatible!):

• ✅ Simulation → simulation-bridge publishes camera data
• ✅ Real Camera → camera-driver publishes camera data
• ✅ Fake Data → fake-camera publishes synthetic data
• ✅ Playback → Log playback publishes recorded data

Control Modes

The node starts in IDLE mode to conserve CPU. Trigger classification via control topic:

Single-Shot (Recommended for UI)

Classify one frame only - perfect for button clicks:

redis-cli PUBLISH "/vision/classification/control" '{"command":"single_shot"}'

Continuous Mode

Start continuous classification at configured rate:

# Start at default rate (from config)
redis-cli PUBLISH "/vision/classification/control" '{"command":"start"}'

# Start at custom rate
redis-cli PUBLISH "/vision/classification/control" '{"command":"start","rate_hz":5}'

# Stop continuous mode
redis-cli PUBLISH "/vision/classification/control" '{"command":"stop"}'

From Behavior Tree

// One-shot classification
ctx.publish("/vision/classification/control", ClassificationControl::SingleShot).await?;

// Wait for result
let result = ctx.receive::<Classification>("/vision/classification").await?;
if result.confidence > 0.8 {
    // High confidence - take action
}

From Dashboard UI

Click "Classify" button → sends single_shot command → shows result

Configuration

File: config/image_classifier.toml

[model]
name = "mobilenet-v2"  # From catalog

[input]
topic = "/robot/sensors/camera/rgb"
processing_rate_hz = 10  # Process 10 frames/second

[output]
topic = "/vision/classification"
top_k = 5  # Return top 5 predictions

Usage

1. Download Model

# Download recommended classification model
mecha10 models pull mobilenet-v2

# Or use setup to get all recommended models
mecha10 setup

2. Run Node

# Standalone
cargo run -p mecha10-nodes-image-classifier

# Or via mecha10 CLI
mecha10 dev --node image-classifier

3. Monitor Classifications

# Watch classification results
mecha10 topics watch /vision/classification

Message Formats

Input: CameraImage

{
  "width": 640,
  "height": 480,
  "format": "jpeg",
  "encoding": "base64",
  "data": "base64_encoded_image_data...",
  "timestamp": 1699459200000
}

Output: Classification

{
  "class": "golden_retriever",
  "class_id": 207,
  "confidence": 0.92,
  "top_k": [
    { "class": "golden_retriever", "class_id": 207, "confidence": 0.92 },
    { "class": "Labrador_retriever", "class_id": 208, "confidence": 0.05 },
    { "class": "cocker_spaniel", "class_id": 219, "confidence": 0.02 },
    { "class": "Irish_setter", "class_id": 172, "confidence": 0.01 },
    { "class": "English_setter", "class_id": 212, "confidence": 0.003 }
  ],
  "timestamp": 1699459200000
}

Models

Supported models from catalog:

• mobilenet-v2 - Fast, 2.5MB (recommended)
• resnet18 - More accurate, 44MB
• Custom ONNX models via model.path

Performance

Target (MacBook Pro M1):

• Latency: ~50ms per frame
• Throughput: 10 FPS (configurable)
• Memory: ~50MB

Integration Example

// Subscribe to classifications in your behavior tree
ctx.subscribe("/vision/classification").await?;

// Use classifications for decision making
if classification.class == "person" && classification.confidence > 0.8 {
    // Execute person-following behavior
}

Development

Build:

cargo build -p mecha10-nodes-image-classifier

Test:

cargo test -p mecha10-nodes-image-classifier

TODO

• Implement actual topic subscription (placeholder loop currently)
• Load full ImageNet 1000 labels
• Load preprocessing params from model catalog
• Add GPU acceleration support
• Add confidence-based frame dropping
• Add temporal smoothing (moving average)