llm_devices
| Crates.io | llm_devices |
| lib.rs | llm_devices |
| version | 0.0.2 |
| source | src |
| created_at | 2024-10-04 19:22:47.251522 |
| updated_at | 2024-10-04 21:41:29.692482 |
| description | Managing Devices and Builds for LLMs |
| homepage | https://github.com/shelbyJenkins/llm_client |
| repository | https://github.com/shelbyJenkins/llm_client |
| max_upload_size | |
| id | 1396998 |
| size | 58,310 |
Shelby Jenkins (ShelbyJenkins)
documentation
README
llm_devices: Managing Devices and Builds for LLMs
This crate is part of the llm_client crate.
The llm_interface crate uses it as a dependency for building llama.cpp.
It's functionality includes:
-
Cloning the specified tag, and building llama.cpp.
-
Checking for device availabilty (CUDA, MacOS) to determine what platform to build for.
-
Fetching available VRAM or system RAM for estimating the correct model to load.
-
Offloading model layers to memory.
-
Logging tools.
See the build documentation for more notes.