ThoughtChain - Autonomous Thought Tracking and Knowledge Management

Persistent reasoning, decision tracking, and dynamic knowledge management for AI systems.

Features

🧠 Autonomous Reasoning (ThoughtChain)

Track AI system's internal reasoning, decisions, and observations across sessions.

• Session-based organization: Group thoughts by user session/project
• Semantic search: Find similar thoughts using vector embeddings
• 6 thought types: Reasoning, Decision, Reflection, Observation, Question, Milestone
• Optional opcode storage: Binary decision encoding for compression
• Full-text search: Fast keyword search via SQLite FTS5

🔍 Conversation Search

Temporal-aware conversation history retrieval with intelligent query detection.

• Temporal context detection: "What did I first ask?", "What did we discuss earlier?"
• Keyword classification: Automatically detect conversation vs knowledge queries
• Semantic + chronological search: Combine embeddings with temporal ordering
• Conversation formatting: Export history for context injection

📚 Dynamic Knowledge Base (Engrams)

Hot-reloadable knowledge archives with filesystem watching.

• Auto-loading: Watch directory and load new .eng or .db files automatically
• Multi-engram search: Query across all loaded knowledge bases simultaneously
• CML format: Structured content with semantic embeddings
• Thread-safe: Concurrent queries with mutex-protected stores

Installation

Add to your Cargo.toml:

[dependencies]
frame-thoughtchain = "0.1.0"

Dependency Architecture

frame-thoughtchain depends on:

frame-thoughtchain
└── frame-catalog (vector search, embeddings, database)

Used by: Frame core for autonomous reasoning

Position in Frame ecosystem:

frame-catalog
    └→ frame-thoughtchain

Quick Start

1. Autonomous Thought Tracking

use frame-thoughtchain::{ThoughtChainStore, ThoughtType};
use sam_memory::database::Database;
use uuid::Uuid;

// Create database and initialize schema
let db = Database::new("thoughts.db")?;
let store = ThoughtChainStore::new(&db);
store.initialize_schema()?;

// Create a session
let session_id = Uuid::new_v4();
store.create_session(session_id, None, Some("my-project"), Some("main"), None)?;

// Log a thought with embedding
let thought_id = store.log_thought(
    session_id,
    ThoughtType::Decision,
    "Decided to use persistent storage for reliability".to_string(),
    vec!["database.rs".to_string()],
    &embedder,
).await?;

// Retrieve recent thoughts
let thoughts = store.get_session_thoughts(session_id, 10)?;
for thought in thoughts {
    println!("{:?}: {}", thought.thought_type, thought.content);
}

// Search by semantic similarity
let query_embedding = embedder.generate("storage decisions")?;
let results = store.search_thoughts(&query_embedding, 5)?;
for (thought, score) in results {
    println!("Similarity {:.3}: {}", score, thought.content);
}

2. Conversation Search

use frame-thoughtchain::{is_conversation_query, search_with_temporal_context};

let query = "What did I first ask about?";

if is_conversation_query(query) {
    // Route to conversation search
    let events = search_with_temporal_context(
        &db,
        conversation_id,
        query,
        &embedder,
        5
    )?;

    let context = format_conversation_context(&events);
    println!("{}", context);
} else {
    // Route to knowledge base search
    // ...
}

3. Dynamic Engram Loading

use frame-thoughtchain::EngramRegistry;

// Create registry with filesystem watching
let registry = EngramRegistry::new("engrams/", true)?;

println!("Loaded {} engrams", registry.list_engrams().len());
println!("Total chunks: {}", registry.total_chunks());

// Search across all engrams
let query_embedding = embedder.generate("How do I use async Rust?")?;
let results = registry.search_all(&query_embedding, 10)?;

for result in results {
    println!("[{}] {:.3}: {}",
        result.engram_id,
        result.score,
        result.content
    );
}

// Add new .eng file to directory → automatically loaded!

API Reference

ThoughtChainStore

Core Methods:

• new(database: &Database) - Create store
• initialize_schema() - Create tables and indices
• create_session() - Start new session
• end_session() - Mark session complete
• log_thought() - Store thought with embedding
• get_session_thoughts() - Retrieve thoughts for session
• get_recent_thoughts() - Get recent thoughts across all sessions
• search_thoughts() - Semantic similarity search

Migration:

• migrate_add_opcode_column() - Add opcode support to existing DB

ThoughtType Enum

pub enum ThoughtType {
    Reasoning,      // Internal analysis
    Decision,       // Choice made + rationale
    Reflection,     // Looking back on past events
    Observation,    // Pattern noticed about user/system
    Question,       // Uncertainty or open question
    Milestone,      // Completion or achievement
}

ThoughtEntry Struct

pub struct ThoughtEntry {
    pub id: Uuid,
    pub session_id: Uuid,
    pub timestamp: DateTime<Utc>,
    pub thought_type: ThoughtType,
    pub content: String,
    pub context: Vec<String>,        // Related files/references
    pub metadata: Option<Value>,     // JSON metadata
    pub opcode_data: Option<Vec<u8>>, // Binary decision encoding
}

Conversation Search Functions

• is_conversation_query(query: &str) -> bool - Detect conversation references
• detect_temporal_context(query: &str) -> TemporalContext - Extract time context
• search_with_temporal_context() - Combined temporal + semantic search
• get_first_user_message() - Get conversation start
• get_recent_messages() - Get last N messages
• format_conversation_context() - Export for context injection

EngramRegistry

Setup:

• new(directory, watch) - Create registry with optional watching
• load_all() - Manually reload all engrams

Querying:

• search_all(embedding, limit) - Query all engrams
• list_engrams() - Get loaded engram IDs
• get_engram_info(id) - Get metadata for specific engram
• total_chunks() - Total chunks across all engrams

Use Cases

1. AI Assistant Long-Term Memory

Track reasoning across sessions to maintain consistency:

// Session 1: User asks about database choice
store.log_thought(
    session_id,
    ThoughtType::Decision,
    "Recommended SQLite for portability and zero-config deployment",
    vec!["database-choice"],
    &embedder
).await?;

// Session 2: User asks why we chose SQLite
let query_embedding = embedder.generate("database recommendation")?;
let thoughts = store.search_thoughts(&query_embedding, 1)?;
// Returns: "Recommended SQLite for portability..."

2. Conversation Context Retrieval

Handle temporal queries naturally:

// "What was the first thing I asked?"
let first_msg = get_first_user_message(&db, conversation_id)?;

// "What did we talk about earlier?"
let context = search_with_temporal_context(
    &db,
    conversation_id,
    "What did we discuss earlier?",
    &embedder,
    5
)?;

3. Hot-Reloadable Knowledge Base

Update knowledge without restarting:

# Application running with engram registry watching "engrams/"
cp new-rust-docs.eng engrams/
# → Automatically detected and loaded
# → Immediately available for queries

4. Milestone Tracking

Track system achievements:

store.log_thought(
    session_id,
    ThoughtType::Milestone,
    "Completed full Frame microservices extraction into 7 focused crates",
    vec!["sam-utils", "frame-thoughtchain", "sam-vector", ...],
    &embedder
).await?;

Database Schema

ThoughtChain uses SQLite with the following tables:

• sessions - User sessions with start/end times
• frame-thoughtchain - Thought entries with full metadata
• frame-thoughtchain_embeddings - Vector embeddings for semantic search
• frame-thoughtchain_fts - FTS5 virtual table for keyword search

Indices on (session_id, timestamp) and (thought_type, timestamp).

Performance

• Thought logging: ~10-50ms (includes embedding generation)
• Semantic search: ~5-20ms for 1000 thoughts
• FTS5 keyword search: <1ms
• Engram hot-reload: ~100-500ms depending on file size
• Memory overhead: ~1KB per thought + embeddings (~1.5KB for 384-dim vectors)

Dependencies

Core:

• rusqlite (0.31) - SQLite with FTS5
• serde, serde_json - Serialization
• chrono - Timestamps
• uuid - Unique identifiers

Engram Support:

• notify (6.0) - Filesystem watching
• cml - Content Markup Language with embeddings

Temporary (until sam-vector extraction):

• sam-memory - Database and embedding generator traits

Future Work

• Extract sam-vector crate (removes sam-memory dependency)
• Opcode compression support (requires sam-opcode crate)
• Multi-user support with access controls
• Thought relationship graph (parent/child thoughts)
• Export thoughts as markdown/JSON
• Thought templates for common patterns

Compatibility

• Rust Edition: 2021
• MSRV: 1.70+
• Platforms: All (platform-independent SQLite)

History

Extracted from the Frame project, where it provides persistent reasoning and knowledge management for the AI assistant.

License

MIT - See LICENSE for details.

Author

Magnus Trent magnus@blackfall.dev

Links

• GitHub: https://github.com/Blackfall-Labs/frame-thoughtchain
• Docs: https://docs.rs/frame-thoughtchain
• Crates.io: https://crates.io/crates/frame-thoughtchain