tokenize-cli
| Crates.io | tokenize-cli |
| lib.rs | tokenize-cli |
| version | 0.1.1 |
| created_at | 2025-06-09 12:25:16.744593+00 |
| updated_at | 2025-06-10 06:46:43.466311+00 |
| description | Tokenize your codebase blazingly fast into a single file for LLM Context 🚀🤖 |
| homepage | |
| repository | https://github.com/DefinitelyNotSimon13/tokenize |
| max_upload_size | |
| id | 1705869 |
| size | 27,879 |
Simon (DefinitelyNotSimon13)
documentation
README
Tokenize
Technically nothing is "tokenized", all files are simply put into one file along with some formatting and an initial prompt.
Usage
Basic Usage
tokenize [TARGET_DIR]
Specify a custom output file
tokenize . -o output.md
[INOP] Specify a custom prompt for the beginning of the file
tokenize . -p prompt.txt
Dont respect .gitignore files
tokenize [TARGET_DIR] --no-gitignore
Include hidden files
tokenize [TARGET_DIR] --include-hidden
Follow Symlinks This was not tested, I am not sure if it actually includes the files in the final output
tokenize [TARGET_DIR] --follow symlinks