archiver

Archiver is a command-line application designed to interact with large language models (LLMs) for testing purposes. It leverages Ollama for model integration and SurrealDB as the vector store. The application uses Langchain-rust as the framework for interacting with the Ollama and SurrealDB.

Prerequisites

Before setting up Archiver, ensure you have the following installed on your system:

• Rust (latest stable version)

• just (command runner)

• Pandoc (load text when adding to vector db)

• Ollama

• SurrealDB

Quick Start

Ensure docker is running and make sure docker compose is setup

docker ps

Install Just to run the setup and Pandocs for file indexing to surrealdb

brew install pandoc just

Once Pandocs and Just are installed run

just docker

Add file to SurrealDB

just add <markdown file>

Or

cargo run -- -a <markdown file>

Add directory of Markdown

cargo run -- -d <directory>

Finally Run Archiver and use it

just run

Change Log

v0.3.2

• Quick Start with Docker: Update to Justfile and Readme to walk through setting up Archiver with docker

v0.3.1

• Check for Pandoc: Ensure pandoc exists before running

v0.3.0

• Index Directory: Add Directory Indexing for bulk Markdown file importing

v0.2.0

• Prompt: Add prompt identifier and colored response from LLM
• Error Handling: Better error handling with SurrealDB and Ollama
• Chuncking: Added document chunking when adding files to SurrealDB
• Mode Switch: Switched LLM chain to a Conversational Retriever from Conversational, making the LLM more efficient with responses from the VectorDB

v0.1.0

• Conversational Memory: Maintain context across interactions with the LLM.
• Vector Stores: Efficient storage and retrieval of high-dimensional vectors through SurrealDB.
• Ollama Integration: Easy setup and use of Ollama for model management.
• Indexing Files: Add Markdown files to SurrealDB.

Install Archiver

1. Install using Cargo:

   cargo install archiver
   

   Make sure you have SurrealDB setup and Ollama before running Archiver

2. Download and Install SurrealDB:

   curl -sSf https://install.surrealdb.com | sh
   

3. Run SurrealDB:

   just db_dev
   

Setting up SurrealDB

Once Surreal is up and running you can use https://surrealist.app/ to access the instance of Surreal. But before doing that a Namespace and Table need to be setup. Fastest way to get this done is to run Archiver once.

1. Set Namespace and Table:

   just run
   

2. Add Markdown File to the SurrealDB:

   cargo run -- -a <markdownfile>
   

Archiver

1. Setup from Source:

   git clone https://github.com/yourusername/archiver.git
   cd archiver
   just db_dev # run in a seperate terminal
   just run
   

License

This project is licensed under the Apache Livense (Version 2.0). See the LICENSE file for details.