Rusty Files

A high-performance, cross-platform file search engine library and CLI written in Rust.

Overview

Rusty Files is a production-grade file indexing and search library designed to be embedded in file managers, IDEs, system utilities, and any application requiring fast file discovery capabilities. It provides:

• Lightning-fast search across millions of files
• Multiple search modes: exact, fuzzy, regex, and glob patterns
• Rich filtering: by extension, size, date, and custom patterns
• Content search: full-text search within files
• Real-time updates: file system watching with automatic index synchronization
• Cross-platform: works on Windows, Linux, and macOS
• Persistent indexing: SQLite-based storage for instant startup
• Simple API: easy to integrate into any Rust project

Features

Core Capabilities

• ✅ Fast Indexing: Multi-threaded directory traversal (10,000+ files/second on SSD)
• ✅ Multiple Search Modes: Exact, case-insensitive, fuzzy, regex, and glob matching
• ✅ Advanced Filtering: Filter by extension, size, modification date
• ✅ Content Search: Full-text search in text files with encoding detection
• ✅ File System Watching: Real-time index updates on file changes
• ✅ Smart Ranking: Results ranked by relevance, recency, and path depth
• ✅ Incremental Updates: Only index changed files
• ✅ Exclusion Rules: Support for .gitignore patterns and custom rules
• ✅ Persistent Index: SQLite-based storage with automatic migrations
• ✅ Thread-safe: Concurrent indexing and searching
• ✅ CLI Application: Full-featured command-line interface

Installation

As a Library

Add to your Cargo.toml:

[dependencies]
srusty-files = "0.1"

As a CLI Tool

cargo install srusty-files

Or build from source:

git clone https://github.com/Aryagorjipour/rusty-files.git
cd rusty-files
cargo build --release

The binary will be available at target/release/filesearch.

Quick Start

Library Usage

use rusty_files::prelude::*;

fn main() -> Result<()> {
    let engine = SearchEngine::new("./index.db")?;

    engine.index_directory("/path/to/search", None)?;

    let results = engine.search("*.rs")?;

    for result in results {
        println!("{}", result.file.path.display());
    }

    Ok(())
}

CLI Usage

filesearch index /path/to/directory

filesearch search "*.rs"

filesearch search "test ext:rs size:>1KB modified:today"

filesearch interactive

filesearch stats

Documentation

Library API

Creating a Search Engine

use rusty_files::prelude::*;

let engine = SearchEngine::builder()
    .index_path("./index.db")
    .thread_count(8)
    .enable_content_search(true)
    .enable_fuzzy_search(true)
    .cache_size(1000)
    .exclusion_patterns(vec![
        ".git".to_string(),
        "node_modules".to_string(),
        "target".to_string(),
    ])
    .build()?;

Indexing Directories

let count = engine.index_directory("/path/to/dir", Some(Box::new(|progress| {
    println!("Indexed: {}/{}", progress.current, progress.total);
})))?;

println!("Indexed {} files", count);

Searching

Simple search:

let results = engine.search("filename")?;

Advanced queries:

use rusty_files::{Query, MatchMode, SizeFilter};

let query = Query::new("test".to_string())
    .with_match_mode(MatchMode::Fuzzy)
    .with_extensions(vec!["rs".to_string(), "toml".to_string()])
    .with_size_filter(SizeFilter::GreaterThan(1024))
    .with_max_results(100);

let results = engine.search_with_query(&query)?;

Query string format:

let results = engine.search("pattern ext:rs,txt size:>1MB modified:today mode:fuzzy")?;

File System Watching

engine.start_watching("/path/to/watch")?;

std::thread::park();

engine.stop_watching()?;

Incremental Updates

let stats = engine.update_index("/path/to/dir", None)?;

println!("Added: {}, Updated: {}, Removed: {}",
    stats.added, stats.updated, stats.removed);

Index Management

let stats = engine.get_stats()?;
println!("Total files: {}", stats.total_files);
println!("Index size: {}", stats.index_size);

let verification = engine.verify_index("/path/to/dir")?;
println!("Health: {:.1}%", verification.health_percentage());

engine.vacuum()?;

engine.clear_index()?;

Query Syntax

The query parser supports the following syntax:

• Basic search: filename
• Extension filter: pattern ext:rs or pattern ext:rs,txt,md
• Size filter:
  • pattern size:>1MB (greater than)
  • pattern size:<500KB (less than)
  • pattern size:1KB..10MB (range)
• Date filter:
  • pattern modified:today
  • pattern modified:yesterday
  • pattern modified:7days or pattern modified:1week
  • pattern modified:>2023-01-01
• Match mode: pattern mode:fuzzy, mode:regex, mode:glob, mode:exact
• Search scope: pattern scope:content, scope:path, scope:name
• Result limit: pattern limit:100

CLI Commands

Index Commands

filesearch index <path>
filesearch index /home/user/projects --progress

filesearch update <path>
filesearch update /home/user/projects --progress

Search Commands

filesearch search "query"

filesearch search "*.rs"

filesearch search "test ext:rs size:>1KB modified:today"

filesearch search "function mode:regex scope:content"

Management Commands

filesearch stats

filesearch verify <path>

filesearch watch <path>

filesearch clear --confirm

filesearch vacuum

Export

filesearch export --output results.json --query "*.rs"

filesearch export --output results.txt --query "test"

Interactive Mode

filesearch interactive

Interactive commands:

• :help - Show help
• :stats - Show index statistics
• :history - Show search history
• :clear - Clear screen
• :quit - Exit

Configuration

Configuration can be loaded from TOML or JSON files:

[config]
index_path = "./filesearch.db"
thread_count = 8
max_file_size_for_content = 10485760  # 10MB
enable_content_search = true
enable_fuzzy_search = true
fuzzy_threshold = 0.7
cache_size = 1000
bloom_filter_capacity = 10000000
bloom_filter_error_rate = 0.0001
max_search_results = 1000
batch_size = 1000
follow_symlinks = false
index_hidden_files = false
exclusion_patterns = [".git", "node_modules", "target", ".DS_Store"]
watch_debounce_ms = 500
enable_access_tracking = true
db_pool_size = 10

Load configuration:

use rusty_files::SearchConfig;

let config = SearchConfig::from_file(&PathBuf::from("config.toml"))?;
let engine = SearchEngine::with_config("./index.db", config)?;

Performance

Benchmarks

Benchmark results from actual runs on the test environment:

Indexing Performance

Search Performance

Test Environment: Linux 4.4.0, Release build with optimizations

Notes:

• Indexing performance scales well with parallel processing
• Search operations are sub-millisecond to low-millisecond range
• Pattern matching is extremely fast (~96 µs)
• Memory usage: <100MB base + configurable cache
• Startup time: <100ms with pre-built index

Optimization Tips

1. Adjust thread count: Set thread_count to CPU cores × 2
2. Tune cache size: Larger cache = faster repeated searches
3. Disable content search: If you don't need it, disable for faster indexing
4. Use exclusion patterns: Skip unnecessary directories
5. Batch operations: Use batch indexing for large directories

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     PUBLIC API LAYER                         │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────────┐    │
│  │SearchEngine│  │ QueryBuilder │  │ ConfigManager    │    │
│  └────────────┘  └──────────────┘  └──────────────────┘    │
└────────────┬────────────────┬────────────────┬──────────────┘
             │                │                │
┌────────────▼────────────────▼────────────────▼──────────────┐
│                    CORE ENGINE LAYER                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │ Indexer  │  │ Searcher │  │ Watcher  │  │ Ranker   │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘   │
└────────────┬────────────┬────────────┬──────────────────────┘
             │            │            │
┌────────────▼────────────▼────────────▼──────────────────────┐
│                   DATA ACCESS LAYER                          │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────────┐    │
│  │ Index DB   │  │ Cache Manager│  │ Bloom Filter     │    │
│  │ (SQLite)   │  │ (LRU)        │  │                  │    │
│  └────────────┘  └──────────────┘  └──────────────────┘    │
└──────────────────────────────────────────────────────────────┘

Examples

See the examples/ directory for complete examples:

• basic_search.rs - Simple file searching
• async_indexing.rs - Asynchronous indexing
• custom_config.rs - Custom configuration

Run examples:

cargo run --example basic_search

Testing

Run tests:

cargo test

cargo test --all-features

cargo test --release

Run benchmarks:

cargo bench

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Built with:

• tokio - Async runtime
• rusqlite - SQLite bindings
• walkdir - Directory traversal
• notify - File system watching
• fuzzy-matcher - Fuzzy matching
• clap - CLI parsing

Support

For issues, questions, or suggestions, please open an issue.