| Crates.io | infiniloom |
| lib.rs | infiniloom |
| version | 0.6.2 |
| created_at | 2025-12-23 04:09:53.466058+00 |
| updated_at | 2026-01-09 12:36:32.293105+00 |
| description | High-performance repository context generator for LLMs - Claude, GPT-4, Gemini optimized |
| homepage | |
| repository | https://github.com/Topos-Labs/infiniloom |
| max_upload_size | |
| id | 2000728 |
| size | 763,759 |
AST-aware code context engine for RAG, vector databases, and AI code assistants
A high-performance Rust library and CLI for generating intelligent code context for LLMs. Uses Tree-sitter AST parsing (22 languages), PageRank symbol ranking, and BLAKE3 content-addressable hashing. Integrates with Pinecone, Weaviate, Qdrant, ChromaDB, and any vector database. Optimized for Claude, GPT-5, GPT-4o, Gemini, Llama, and 27+ LLM tokenizers.
# Install
npm install -g infiniloom
# Generate AI-ready context in one command
infiniloom pack . --output context.xml
# → Paste into Claude, GPT, or any LLM
# Generate chunks for your vector database
infiniloom embed . -o chunks.jsonl
# → Import into Pinecone, Weaviate, Qdrant, etc.
When you ask an AI to help with code, quality depends almost entirely on what context you provide. Most approaches fail:
The result: AI gives generic answers, hallucinates function signatures, or misses critical dependencies — not because the AI is bad, but because the context is bad.
Infiniloom reads your codebase and produces a structured summary designed specifically for AI consumption.
Think of it like this: instead of handing someone a filing cabinet and saying "figure it out," you give them a well-organized briefing document that highlights what matters.
Here's how it works:
Analyzes structure — Infiniloom understands how your code is organized: which files exist, how they relate, what languages are used.
Extracts meaning — It identifies the important pieces: functions, classes, interfaces, types. Not just text, but semantic units that AI can reason about.
Ranks importance — Using techniques similar to how search engines rank web pages, it determines which code is central to your project and which is peripheral.
Filters noise — Generated files, build artifacts, vendored dependencies, and other distractions are automatically excluded.
Formats for AI — The output is structured in ways that different AI models understand best — XML for Claude, Markdown for GPT-4o/GPT-5, YAML for Gemini.
The result is a context package that helps AI give you accurate, relevant answers about your actual code.
Infiniloom's embed command generates deterministic, content-addressable code chunks designed specifically for retrieval-augmented generation:
# Generate chunks for your vector database
infiniloom embed ./my-repo -o chunks.jsonl
# Only get changed chunks (incremental updates)
infiniloom embed ./my-repo --diff -o updates.jsonl
| Feature | Benefit |
|---|---|
| Content-Addressable IDs | Same code anywhere produces same ID (ec_a1b2c3d4...) — enables cross-repo deduplication |
| AST-Aware Chunking | Never splits mid-function or mid-class — preserves semantic boundaries |
| Incremental Updates | Manifest-based diffing detects added/modified/removed chunks — only re-embed what changed |
| Hierarchical Chunks | Parent-child relationships preserved — container summaries link to member chunks |
| Auto-Generated Tags | Semantic tags (async, security, database, http) improve retrieval relevance |
| Call Graph Context | calls and called_by fields enable dependency-aware retrieval |
Works with any vector database that accepts JSON/JSONL:
# Pinecone / Weaviate / Qdrant
infiniloom embed . --max-tokens 1500 -o chunks.jsonl
# Import chunks.jsonl using your vector DB's bulk import
# ChromaDB / pgvector / Milvus
infiniloom embed . --format json -o chunks.json
# Parse JSON array and insert with your preferred client
Each chunk includes rich metadata for filtering and retrieval:
{
"id": "ec_a1b2c3d4e5f6g7h8",
"content": "async fn authenticate(token: &str) -> Result<User, AuthError> {...}",
"tokens": 245,
"kind": "function",
"source": {
"file": "src/auth.rs",
"symbol": "authenticate",
"fqn": "src::auth::authenticate",
"language": "Rust"
},
"context": {
"docstring": "Validates JWT token and returns authenticated user",
"calls": ["verify_jwt", "find_user_by_id"],
"called_by": ["login_handler", "refresh_token"],
"tags": ["async", "security", "public-api"]
}
}
See the embed command documentation for complete details.
Install:
npm install -g infiniloom
Generate context for your repository:
infiniloom pack . --output context.xml
This produces an XML file containing your codebase's structure, key symbols, and content — ready to paste into Claude, GPT, or any other AI assistant.
| Capability | Why It Matters |
|---|---|
| Repository analysis | Understands project structure, languages, and file relationships automatically |
| Symbol extraction | Identifies functions, classes, and types — the units AI reasons about |
| Importance ranking | Highlights central code, deprioritizes utilities and boilerplate |
| Noise reduction | Excludes generated files, dependencies, and artifacts by default |
| Security filtering | Detects and redacts API keys, tokens, and credentials before they reach AI |
| Multiple output formats | XML, Markdown, YAML, JSON — optimized for different AI models |
| Token-aware packaging | Respects context limits so you can fit within model constraints |
| Git integration | Understands diffs, branches, and commit history for change-aware context |
| 22 language support | Full parsing for Python, JavaScript, TypeScript, Rust, Go, Java, C/C++, and more |
| Command | What It Does |
|---|---|
pack |
Analyze a repository and generate AI-ready context |
scan |
Show repository statistics: files, tokens, languages |
map |
Generate a ranked overview of key symbols |
embed |
Generate chunks for vector databases / RAG systems |
diff |
Build context focused on recent changes |
index |
Create a symbol index for fast queries |
impact |
Analyze what depends on a file or function |
chunk |
Split large repositories for multi-turn conversations |
init |
Create a configuration file |
See the Command Reference for detailed documentation.
| Feature | Benefit |
|---|---|
| 🎯 Smart Ranking | PageRank algorithm identifies important symbols — prioritizes core business logic over utilities |
| 🔗 Content-Addressable | BLAKE3 hashing produces stable chunk IDs — same code anywhere = same ID for deduplication |
| 🌳 AST-Aware | Tree-sitter parsing (22 languages) preserves semantic boundaries — never splits mid-function |
| 🔒 Security-First | Automatic secret detection with regex + NFKC normalization prevents API key leaks |
| 📊 27+ Tokenizers | Exact counts for GPT-5/4o via tiktoken, calibrated estimation for Claude/Gemini/Llama |
| 🚀 Blazing Fast | Pure Rust + Rayon parallelism — handles 100K+ file repos in seconds |
| 🔄 Incremental | Manifest-based diffing tracks added/modified/removed chunks — only re-embed what changed |
| 📈 Call Graphs | calls and called_by fields enable dependency-aware retrieval |
| Feature | Infiniloom | Repomix | Aider | Continue | Cursor |
|---|---|---|---|---|---|
| AST Parsing (Tree-sitter) | ✅ 22 languages | ❌ | ❌ | ❌ | ✅ |
| PageRank Symbol Ranking | ✅ | ❌ | ❌ | ❌ | ❌ |
| Content-Addressable Chunks | ✅ BLAKE3 | ❌ | ❌ | ❌ | ❌ |
| Incremental Updates (Diffing) | ✅ Manifest-based | ❌ | ✅ Git-based | ❌ | ✅ |
| Secret Detection/Redaction | ✅ 15+ patterns | ❌ | ❌ | ❌ | ❌ |
| Multi-Model Token Counting | ✅ 27 models | ❌ | ✅ Few models | ❌ | ❌ |
| Call Graph Extraction | ✅ | ❌ | ❌ | ❌ | ❌ |
| Vector DB Integration | ✅ Native JSONL | ❌ | ❌ | ❌ | ❌ |
| Hierarchical Chunking | ✅ | ❌ | ❌ | ❌ | ❌ |
| CLI Tool | ✅ | ✅ | ✅ | ❌ | ❌ |
| Library (Rust/Python/Node) | ✅ | ✅ | ❌ | ❌ | ❌ |
| IDE Integration | 🔜 Coming | ❌ | ❌ | ✅ Native | ✅ Native |
| Price | Free/OSS | Free/OSS | Free/OSS | Free tier | $20/mo |
| Tool | Best For | Not Ideal For |
|---|---|---|
| Infiniloom | RAG pipelines, vector DBs, security-conscious teams, large codebases, CI/CD automation | Real-time IDE completions |
| Repomix | Quick one-off context dumps, small projects | Large repos, incremental updates, security |
| Aider | Interactive pair programming, git-based workflows | Headless automation, RAG systems |
| Continue.dev | IDE code completion, inline suggestions | Batch processing, RAG pipelines |
| Cursor | Full AI-powered development environment | Headless/CLI workflows, custom pipelines |
Compared to "just paste the code":
Infiniloom understands code structure. It knows the difference between a core business function and a utility helper. It understands imports, dependencies, and relationships. Pasting files gives AI text; Infiniloom gives AI understanding.
Compared to generic RAG tools:
Most RAG systems treat code as documents. They chunk by character count, embed text, and retrieve by similarity. This misses the structure that makes code meaningful. Infiniloom preserves semantic boundaries — functions stay whole, relationships stay intact.
Compared to embedding-based approaches:
Embeddings are useful for "find code similar to X." They're less useful for "understand how X works." Infiniloom focuses on comprehension: what exists, how it connects, what matters. This is about building complete context, not searching fragments.
Our philosophy:
Context quality beats context quantity. A smaller, well-structured context produces better AI responses than a larger, noisier one. Infiniloom prioritizes signal over volume.
Good fit:
Probably not needed:
Infiniloom is stable and actively maintained.
What's solid today:
embed command for vector database chunkingComing next:
| Method | Command |
|---|---|
| npm (recommended) | npm install -g infiniloom |
| Homebrew (macOS) | brew tap Topos-Labs/infiniloom && brew install --cask infiniloom |
| Cargo (Rust users) | cargo install infiniloom |
| pip (Python library) | pip install infiniloom |
| From source | git clone https://github.com/Topos-Labs/infiniloom && cd infiniloom && cargo build --release |
Infiniloom supports tab completion for bash, zsh, fish, PowerShell, and Elvish.
infiniloom completions bash > /tmp/infiniloom.bash
sudo mv /tmp/infiniloom.bash /etc/bash_completion.d/
infiniloom completions zsh > ~/.zfunc/_infiniloom
# Add to ~/.zshrc:
fpath=(~/.zfunc $fpath)
autoload -U compinit && compinit
infiniloom completions fish > ~/.config/fish/completions/infiniloom.fish
infiniloom completions powershell | Out-String | Invoke-Expression
# Or add to your profile:
infiniloom completions powershell >> $PROFILE
infiniloom completions elvish > ~/.config/elvish/completions/infiniloom.elv
We welcome contributions of all kinds: bug reports, feature requests, documentation improvements, and code.
cargo test --workspace # Run tests
cargo clippy --workspace # Lint
cargo fmt --all # Format
MIT — see LICENSE.
Made by Topos Labs