github code searcher

A powerful, concurrent CLI tool for searching code across GitHub repositories with advanced rate-limit handling and progress visualization.

Features

• Efficient Code Searching: Search GitHub's codebase for specific keywords or phrases
• Concurrent Execution: Run multiple searches in parallel with configurable concurrency
• Smart Rate-Limit Handling: Automatically detects and waits for GitHub API rate limits with visual feedback
• Detailed Progress Tracking: Real-time progress indicators for each search term
• Pagination Support: Configure maximum pages per search term
• JSON Output: Structured output format for post-processing

Installation

Prerequisites

• Rust and Cargo installed (Install Rust)
• GitHub Personal Access Token (Create token)

Building from Source

# Clone the repository
git clone https://github.com/manorajesh/github-code-searching.git
cd github-code-searching

# Build the project
cargo build --release

# The binary will be available at
# ./target/release/github-code-searching

Usage

# Basic usage
github-code-searching --words "rust concurrency" "tokio async" --token YOUR_GITHUB_TOKEN

# Search with environment variable for token
export GITHUB_TOKEN=your_github_token
github-code-searching -w "axum router" "rocket framework"

# Control concurrency and output location
github-code-searching -w "security vulnerability" "authentication bypass" -c 3 -o security_findings.json

# Limit pages per search term
github-code-searching -w "kubernetes operator" -p 5

Command-Line Arguments

Authentication

The tool requires a GitHub token with appropriate permissions:

1. Use the --token argument to provide your token directly
2. Set the GITHUB_TOKEN environment variable
3. Create a .env file in the project directory with GITHUB_TOKEN=your_token

Output Format

Results are saved in JSON format with the following structure for each match:

{
  "name": "filename.ext",
  "html_url": "https://github.com/owner/repo/blob/...",
  "sha": "commit-hash",
  "search_term": "search-word",
  "repository_owner": {
    "login": "owner-name",
    "avatar_url": "https://avatars.githubusercontent.com/...",
    "html_url": "https://github.com/owner"
  },
  "text_matches": [
    {
      "object_url": "https://api.github.com/...",
      "object_type": "FileContent",
      "property": "content",
      "fragment": "code fragment containing match",
      "matches": [
        {
          "text": "matched text",
          "indices": [start, end]
        }
      ]
    }
  ]
}

How It Works

GitHub Code Searcher performs the following operations:

1. Parses command-line arguments and authenticates with GitHub
2. Creates progress indicators for each search term
3. Spawns concurrent search tasks limited by the specified concurrency
4. Each search task handles pagination through search results
5. Monitors and respects GitHub API rate limits with visual feedback
6. Filters and saves relevant search results to the specified output file

Rate Limit Handling

The tool includes sophisticated rate-limit management:

• Monitors remaining API calls through GitHub response headers
• Displays real-time rate limit status
• Automatically pauses with countdown when limits are reached
• Resumes operations when rate limits reset

Development

Project Structure

• src/main.rs: Entry point and CLI argument parsing
• src/github_searcher.rs: Core search functionality and API interaction

Building and Testing

# Run with debug output
RUST_LOG=debug cargo run -- -w "example search term"

# Run tests
cargo test

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.