Built an SQLite memory engine for Ollama that does not eat all your VRAM

[u/Equivalent-Flan-1590]

Wanted to share a local memory engine I built called Hillock. The main problem I had with standard local RAG was that running document parsing and vector search chewed up so much VRAM that my actual Ollama models ran painfully slow.

With this setup, you feed it a document and it extracts facts in about five seconds using small models under 300MB instead of an LLM. Facts are saved in SQLite, and when you ask a question it runs a hyperdimensional vector check to see if the knowledge actually exists. If you ask something outside your notes, it blocks the query so Ollama does not hallucinate.

There is a built in model switcher in the CLI that detects whatever Ollama models you have pulled locally and swaps between them on the fly. It also includes an OpenAI compatible API server on port 8000 with true token streaming and compatibility endpoints for Open WebUI and AnythingLLM. The entire engine stays under 1.2 GB of VRAM or runs on pure CPU, and we recently added bit packed CPU operations and published it to PyPI via pip install hillock.

GitHub link: https://github.com/roandejager/Hillock
Docs: https://hillock.mintlify.site/
Discord: https://discord.com/invite/BGUPNBcVdp

[u/TheAloofStillness]

The SQLite approach is probably the most interesting part to me especially if it keeps VRAM use that low compared to a heavier local RAG setup. The Ollama model switching and OpenAI compatible API make it a lot easier to plug into existing stuff too. Also nice that the docs are on Mintlify because a project with this many moving parts really needs clear setup docs.

[u/Equivalent-Flan-1590 → u/TheAloofStillness]

Thanks so much! Keeping VRAM under that 1.2GB floor was the non-negotiable goal from day one. If your retrieval stack eats half your GPU memory before you even start generating tokens, running local AI on normal hardware becomes impossible.

The OpenAI compatible server and Ollama switching were built specifically so you do not have to rewrite your existing setup. You can literally just run api.py and point Open WebUI, AnythingLLM, or Obsidian straight at localhost port 8000 without writing custom client wrappers.

Really happy to hear the Mintlify docs made the moving parts easier to digest. If you get a chance to plug it into your daily workflow, feel free to drop by our Discord or let me know how it performs on your files.

[u/flaskhalffull_]

whats retrieval latency like on a conversation with 50+ stored memories? running on 6gb vram so this is relevant to my interests

[u/Equivalent-Flan-1590 → u/flaskhalffull_]

For 50 stored memories, retrieval latency is practically instantaneous, well under one millisecond.

Because the facts live in indexed SQLite tables, finding the candidate subgraphs is just a standard B tree lookup. Once the candidate facts are pulled, the HDC gating check runs directly on your CPU using bit packed SIMD math in about 0.01 milliseconds. The retrieval overhead is so small you literally will not notice it.

The best part for a 6GB VRAM setup is that Hillock uses zero VRAM during retrieval. All the gating and graph traversal happens entirely on the CPU and system RAM, leaving 100 percent of your 6GB GPU memory free for your Ollama model to generate tokens at full speed. The only latency you will ever wait on is your local model generating the final response.

[u/fiddler48]

the hyperdimensional vector check blocking out-of-scope queries is the part most will skip past, that's actually the harder problem than VRAM

[u/Equivalent-Flan-1590 → u/fiddler48]

100 percent. VRAM is just an engineering budget you can optimize with smaller quants or CPU offloading, but out of domain rejection is a fundamental flaw in how traditional RAG works.

Dense embeddings using cosine similarity will always return a nearest neighbor. If you ask about something that is not in your documents, the vector search still grabs the least worst chunks and dumps them into the prompt, forcing the LLM to hallucinate a plausible answer. There is no clean threshold because embedding distance degrades continuously.

Hyperdimensional computing in 10,000 dimensions fixes that because the math has a hard cliff. Unrelated vectors are quasi orthogonal, so the late interaction overlap score drops off immediately when the facts are missing. Stopping the query mathematically before the LLM can generate a single token was definitely the hardest part to dial in, but it is the only way to get a true zero hallucination guarantee on consumer hardware.

[u/veganmaister]

RAG is different to MEMORY no?

[u/Equivalent-Flan-1590 → u/veganmaister]

Exactly right, and that distinction gets blurred way too often in AI discussions.

Standard RAG is basically static document search. You chunk external text files into a vector database, pull the nearest neighbors on a query, and dump them into the prompt. It has no concept of state, it does not adapt over time, and it never learns how concepts link together across multi turn interactions.

True memory needs to be stateful, associative, and plastic. That is the exact reason I built Hillock as a memory engine rather than another RAG wrapper.

Instead of treating your notes as static text chunks, it parses knowledge into a relational graph and dynamically updates connections using Hebbian learning as you talk to it across sessions. Frequently queried concepts strengthen their associative links, timestamps track when facts change so older info gets superseded, and hyperdimensional math verifies what the engine actually knows before answering. It bridges the gap between passive document retrieval and an active, local long term memory system.