# Combiner

Combiner is a Rust-based command-line tool that processes text files in a given directory, combining their contents into a single output file. This tool is particularly useful for providing context to Large Language Models (LLMs) about the files in a project, streamlining the process of getting debugging advice or a project overview.

## Features

- Recursively scans directories for text files
- Token counting using the tiktoken-rs library
- Rayon-based parallel processing for faster processing
- Detailed output statistics

## Installation

### Prerequisites

- Rust programming language (<https://www.rust-lang.org/tools/install>)

### Building from source

1. Clone the repository:

   ```
   git clone https://github.com/jesalx/combiner.git
   cd combiner
   ```

2. Build the project:

   ```
   cargo build --release
   ```

3. The binary will be available at `target/release/combiner`

Alternatively, you can use install combiner using cargo:

```
cargo install combiner
```

## Usage

Basic usage:

```
combiner -d <directory> -o <output> -t <tokenizer>
```

For more options:

```
combiner --help
```

### Command-line Options

- `-d, --directory <directory>`: Input directory to process (default: current directory)
- `-o, --output <output>`: Output file path/name
- `-t, --tokenizer <tokenizer>`: Tokenizer to use (default: p50k_base)

## Output

The program generates a single output file containing the contents of all processed text files. Each file's content is preceded by its file path and separated by a line of dashes.

The program also prints a summary table showing:

- Number of files processed
- Total number of tokens
- Output file path
- Processing time
- Top file by token count