valentinus
| Crates.io | valentinus |
| lib.rs | valentinus |
| version | 0.5.5 |
| source | src |
| created_at | 2024-07-12 12:34:04.957532 |
| updated_at | 2024-12-06 11:44:27.827383 |
| description | Next generation vector database built with LMDB bindings |
| homepage | |
| repository | https://github.com/kn0sys/valentinus |
| max_upload_size | |
| id | 1300631 |
| size | 311,387 |
n12n (kn0sys)
documentation
https://docs.rs/valentinus
README
valentinus
next generation vector db built with lmdb bindings
dependencies
- bincode/serde - serialize/deserialize
- lmdb-rs - database bindings
- ndarray - numpy equivalent
- ort/onnx - embeddings
getting started
git clone https://github.com/kn0sys/valentinus && cd valentinus
optional environment variables
| var | usage | default |
|---|---|---|
LMDB_USER |
working directory of the user for database | $USER |
LMDB_MAP_SIZE |
Sets max environment size, i.e. size in memory/disk of all data | 20% of available memory |
ONNX_PARALLEL_THREADS |
parallel execution mode for this session | 1 |
VALENTINUS_CUSTOM_DIM |
embeddings dimensions for custom models | all-mini-lm-6 -> 384 |
VALENTINUS_LMDB_ENV |
environment for the database (i.e. test, prod) | test |
tests
- Note: all tests currently require the
all-MiniLM-L6-v2_onnxdirectory - Get the model.onnx and tokenizer.json from huggingface or build them
mkdir all-MiniLM-L6-v2_onnx
cd all-MiniLM-L6-v2_onnx && wget https://huggingface.co/nigel-christian/all-MiniLM-L6-v2_onnx/resolve/main/config.json
wget https://huggingface.co/nigel-christian/all-MiniLM-L6-v2_onnx/resolve/main/model.onnx
wget https://huggingface.co/nigel-christian/all-MiniLM-L6-v2_onnx/resolve/main/special_tokens_map.json
wget https://huggingface.co/nigel-christian/all-MiniLM-L6-v2_onnx/resolve/main/tokenizer_config.json
wget https://huggingface.co/nigel-christian/all-MiniLM-L6-v2_onnx/resolve/main/tokenizer.json
wget https://huggingface.co/nigel-christian/all-MiniLM-L6-v2_onnx/resolve/main/vocab.txt
RUST_TEST_THREADS=1 cargo test
examples
see examples