llm_devices
| Crates.io | llm_devices |
| lib.rs | llm_devices |
| version | 0.0.3 |
| created_at | 2024-10-04 19:22:47.251522+00 |
| updated_at | 2025-01-29 01:26:24.671944+00 |
| description | llm_devices: Device management and build system for LLM inference |
| homepage | https://github.com/shelbyJenkins/llm_client |
| repository | https://github.com/shelbyJenkins/llm_client |
| max_upload_size | |
| id | 1396998 |
| size | 69,808 |
Shelby Jenkins (ShelbyJenkins)
documentation
README
llm_devices: Device management and build system for LLM inference
The llm_devices crate is a workspace member of the llm_client project. It is used as a dependency by the llm_interface crate for building llama.cpp.
Features
- Automated building of llama.cpp with appropriate platform-specific optimizations
- Device detection and configuration for CPU, RAM, CUDA (Linux/Windows), and Metal (macOS)
- Manages memory by detecting available VRAM/RAM, estimating model fit, and distributing layers across devices
- Logging tools