Terminator Workflow Recorder

A comprehensive workflow recording library for Windows that captures user interactions with UI elements, including mouse clicks, keyboard input, clipboard operations, and UI automation events.

Features

• Input Recording: Mouse movements, clicks, keyboard input
• UI Element Capture: Detailed information about UI elements being interacted with
• Clipboard Monitoring: Track copy/paste operations
• Hotkey Detection: Record keyboard shortcuts and hotkey combinations
• UI Automation Events: Focus changes, property changes, structure changes
• Noise Filtering: Built-in filtering to ignore system UI noise like clock updates

Usage

Basic Recording

use terminator_workflow_recorder::{WorkflowRecorder, WorkflowRecorderConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = WorkflowRecorderConfig::default();
    let mut recorder = WorkflowRecorder::new("My Workflow".to_string(), config);

    recorder.start().await?;

    // ... perform your workflow ...

    recorder.stop().await?;
    recorder.save("workflow.json")?;

    Ok(())
}

Filtering System UI Noise

The recorder includes built-in filtering to ignore noisy system UI elements like the clock, notifications, and other system components. You can customize this filtering:

let config = WorkflowRecorderConfig {
    // Enable UI event recording
    record_ui_focus_changes: true,
    record_ui_property_changes: true,

    // Filter out noisy system elements
    ignore_focus_patterns: vec![
        "clock".to_string(),
        "notification".to_string(),
        "tooltip".to_string(),
        "popup".to_string(),
    ],
    ignore_property_patterns: vec![
        "clock".to_string(),
        "time".to_string(),
        "pm".to_string(),
        "am".to_string(),
    ],
    ignore_window_titles: vec![
        "Windows Security".to_string(),
        "Action Center".to_string(),
        "Task Manager".to_string(),
    ],
    ignore_applications: vec![
        "explorer.exe".to_string(),
        "dwm.exe".to_string(),
        "winlogon.exe".to_string(),
    ],

    // Other configuration options
    ..Default::default()
};

Configuration Options

Recording Controls

• record_mouse: Enable/disable mouse event recording
• record_keyboard: Enable/disable keyboard event recording
• record_clipboard: Enable/disable clipboard operation recording
• record_ui_focus_changes: Enable/disable UI focus change events
• record_ui_property_changes: Enable/disable UI property change events

Noise Reduction

• mouse_move_throttle_ms: Minimum time between mouse move events (default: 50ms)
• ignore_focus_patterns: Patterns to ignore in focus change events
• ignore_property_patterns: Patterns to ignore in property change events
• ignore_window_titles: Window titles to ignore for all UI events
• ignore_applications: Application names to ignore for all UI events

Content Limits

• max_clipboard_content_length: Maximum clipboard content to record (default: 1KB)
• max_text_selection_length: Maximum text selection length to record (default: 512 chars)

Common Filtering Patterns

Clock and Time Elements

ignore_property_patterns: vec![
    "clock".to_string(),
    "time".to_string(),
    "pm".to_string(),
    "am".to_string(),
],

System Notifications

ignore_focus_patterns: vec![
    "notification".to_string(),
    "action center".to_string(),
    "toast".to_string(),
],

Taskbar and System Tray

ignore_focus_patterns: vec![
    "taskbar".to_string(),
    "system tray".to_string(),
    "start button".to_string(),
],

Windows System Applications

ignore_applications: vec![
    "dwm.exe".to_string(),           // Desktop Window Manager
    "explorer.exe".to_string(),      // Windows Explorer
    "winlogon.exe".to_string(),      // Windows Logon
    "csrss.exe".to_string(),         // Client Server Runtime
],

Double Click Detection

The workflow recorder now supports double click detection with the following features:

Features

• Automatic Detection: Detects double clicks based on timing and position thresholds
• Configurable Thresholds:
  • Time threshold: 500ms (Windows standard)
  • Distance threshold: 5 pixels tolerance
• Button Support: Works with all mouse buttons (Left, Right, Middle)
• UI Element Capture: Captures the UI element that was double-clicked

Configuration

let config = WorkflowRecorderConfig {
    capture_ui_elements: true,  // Enable to capture UI elements on double clicks
    // ... other settings
};

Event Structure

Double clicks generate WorkflowEvent::Mouse events with MouseEventType::DoubleClick:

match event {
    WorkflowEvent::Mouse(mouse_event) => {
        match mouse_event.event_type {
            MouseEventType::DoubleClick => {
                println!("Double click at ({}, {})",
                    mouse_event.position.x,
                    mouse_event.position.y);

                if let Some(element) = &mouse_event.metadata.ui_element {
                    println!("Element: {} ({})",
                        element.name_or_empty(),
                        element.role());
                }
            }
            _ => {}
        }
    }
    _ => {}
}

Example Usage

See examples/double_click_demo.rs for a complete example:

cargo run --example double_click_demo

This demo will:

• Start recording mouse events
• Detect and log double clicks with position and UI element information
• Show timing and distance-based filtering in action

Testing

The implementation includes comprehensive unit tests:

cargo test test_double_click_tracker

Tests cover:

• Basic double click detection
• Timing threshold enforcement (500ms)
• Distance threshold enforcement (5 pixels)
• Different button handling
• Tracker reset functionality

Output Format

The recorder saves workflows as JSON files containing timestamped events:

{
  "name": "My Workflow",
  "start_time": 1748456891489,
  "end_time": 1748456956367,
  "events": [
    {
      "timestamp": 1748456891524,
      "event": {
        "Keyboard": {
          "key_code": 65,
          "is_key_down": true,
          "character": "a",
          "metadata": {
            "ui_element": {
              "role": "textfield",
              "name": "Search Box"
            }
          }
        }
      }
    }
  ]
}

Performance Considerations

• Use filtering to reduce event volume for better performance
• Consider disabling UI automation events (record_ui_*) if not needed
• Adjust mouse_move_throttle_ms to balance accuracy vs. performance
• Set appropriate content length limits for clipboard and text selection

Platform Support

Currently supports Windows only. Requires Windows 10/11 with UI Automation support.