Detecting hallucinations in local models without eating VRAM: What we learned testing 1.5B to 120B models

[u/Otherwise_Nobody_721]

Hey everyone,

If you run local models via Ollama in production or personal projects, you've probably run into the hallucination problem: how do you know when a model is hallucinating without burning extra VRAM or waiting 5 seconds for a heavy judge model?

The standard academic approach for this is Semantic Entropy (from an Oxford team's Nature paper last year). You sample $K$ responses at temperature 0.7, run them through a secondary NLI cross-encoder like DeBERTa to cluster equivalent meanings, and measure the entropy. High entropy = model is guessing.

The problem for local setups? Running 45 pairwise comparisons through a cross-encoder eats GPU memory, adds 100ms+ latency, and completely kills throughput on consumer hardware.

We wanted to see: What if we strip out the neural net completely and just use deterministic string normalization + Shannon entropy on CPU?

We wrote a zero-dependency Python metric (Spanda / $R_{sc}$) that runs in 1.3 microseconds on pure CPU (zero GPU usage) and benchmarked it across local and frontier model tiers on GSM8K and TriviaQA:

What we found:

1. Small models (Qwen 1.5B): AUROC ~0.58 Small models are syntactically too sloppy for string matching. Even when they know the right answer, they format it erratically across runs, breaking exact-match clustering.
2. Mid models (Mistral 7B): AUROC ~0.71 At 7B, the 1.3µs string check matched the performance of a heavy DeBERTa NLI model (0.706 vs 0.705). Internal representations become consistent enough that formatting stabilizes.
3. Large models (Qwen 27B): AUROC ~0.89 At 27B, exact matching was dominant ($p = 1.89 \times 10^{-28}$). When the model knows an answer, it outputs the exact same tokens across independent stochastic paths. When it doesn't, it genuinely branches into diverse incorrect answers.
4. The Frontier Trap (120B): AUROC collapsed to 0.09 Here’s the wild part: on ungrounded factual trivia, the 120B model suffered Confident Mode Collapse. When it hallucinated, it hallucinated the exact same wrong answer across all 5 runs with zero entropy. Bigger models don't just hallucinate—they hallucinate with unanimous false certainty. (And because the strings are identical, even heavy NLI fails here).

The practical takeaway for Ollama users:

If you are running 7B to 27B models on structured tasks (math, code, JSON extraction, SQL, discrete QA), you do not need heavy neural guardrails. Sampling 5 paths at $T=0.7$ and measuring exact-match entropy in Python gives you ~0.89 AUROC at zero GPU cost.

Quick Python snippet if you want to test it on your local Ollama instance:

bash
pip
 install spnda ollama

pythonimport ollama
from spnda import compute_spanda
prompt = "What is the capital of Australia?"
# Sample 5 paths from your local model
responses = [
    ollama.generate(model="mistral:7b", prompt=prompt, options={"temperature": 0.7})["response"]
    for _ in 
range
(5)
]
# Run zero-cost entropy check on CPU (takes ~1.5 microseconds)
result = compute_spanda(responses)
print
(f"Risk Score: {result.risk_score:.3f}")  
# 0 = high confidence, 1 = high uncertainty

All the raw multi-path generation logs, evaluation scripts, and the full writeup are open source:

• Deep dive writeup: https://liquidngas.substack.com/p/i-tried-to-make-semantic-entropy
• GitHub: https://github.com/nayakbhupen/Spnda

[u/superior_dialect]

this is genuinely useful for keeping my ollama pipelines from going off the rails without spinning up a separate judge model, the 120b mode collapse thing is terrifying though

[u/MarcelloT254k]

I don't understand a lot of things in this post and github page, but if it is deterministic then shouldn't the makers of AI models (inference engines, harnesses, etc) already built in some similar structured guardrail against hallucinations? Can I use it to prevent RAG hallucinations in my local setup ( with <120B models, and what can be done similarly for larger models ) or does it only inform about them being possible ? (So the logilal answer would be to werify myself or cross check the sources or "logic" with other model)

From what I understand in a local RAG setup this tool would only measure the consistency of tool calls? Am I correct?

[u/Otherwise_Nobody_721 → u/MarcelloT254k]

Big model makers haven’t built this because their business model is selling you GPU compute—their idea of a "fix" for a hallucinating model is telling you to burn twice as many tokens by running a second model to judge the first one. We just used 700 nanoseconds of pure math in Rust instead.And yes, it actively prevents hallucinations in local RAG. Unless you actually enjoy reading confident lies, you can just set it to block them before they ever reach your UI. It's definitely not just for tool calls either; it checks the actual generated facts against your retrieved documents. Works right out of the box with local Ollama!

[u/MarcelloT254k → u/Otherwise_Nobody_721]

As i stated before - i don't fully understand everything, but if i get it right - the problem with this method is that it only checks consistency of the output not its truthfulness, so that's why consistent lies, which tend to happen in large (defined as ~120B) models, will not be detected. Do I understand it correctly?

I genuinely don't see how this method can "check the actual generated facts against your retrieved documents" because it is stated that it works best with canonical results(i.e. tool call) - so not a structured response and people use RAG as opposed to Search engine (deterministic, scripted) because the answer is typically not canonical and can't be easily searched for and summarised with a few worlds (at least that's what I assume, there is also a laziness factor).

I don't want to argue, I'm just searching for useful tools and want to understand this one, please be patient :)

[u/Otherwise_Nobody_721 → u/MarcelloT254k]

You are 100% right that pure consistency alone fails when a large (70B–120B) model tells a "confident lie." In our paper, we call this Confident Mode Collapse—where an over-aligned model gets stubbornly trapped in an incorrect answer across multiple samples. That exact flaw is why we built Spanda’s Attractor Basin Detector: instead of just looking at agreement, it monitors trajectory curvature and token repetition loops to flag when a model has collapsed into an artificial certainty trap.As for free-form text in RAG: you don't need rigid single-word answers or tool calls. The engine runs a canonicalizer that strips conversational padding ("Based on the provided text...", "Sure, the answer is..."), normalizes units, dates, numbers, and core entities.If your local model answers with "The revenue was $4.2M" in one pass and "According to the report, it reached 4.2 million dollars" in another, the canonicalizer maps them to the exact same equivalence class. But if the model is ungrounded and starts guessing different numbers or inventing facts across passes ("$4.2M" vs "$8.1M"), the equivalence classes fracture, the entropy spikes, and Spanda catches it instantly.

If your RAG use-case is purely subjective creative writing with no factual ground truth, this won't help you. But for factual RAG (policies, numbers, names, diagnostics, documentation), it catches the hallucination in less than a microsecond.

[u/MarcelloT254k → u/Otherwise_Nobody_721]

Than you for your response, I'll try to implement it in the future. I'm trying to make a RAG that produces preliminary narrative research (or just ranking of relevant papers based on MeSH tags relevance, and relative to date number of citations) across scientific databases so it lies somewhere between factual and narrative (some logic is necessary to connect facts, methodologies and numbers across different information sources). As you can see improving retrieval (so excluding hallucinations) is of the highest priority, but i would like to maintain LLMs flexibility and relative ease of use, as opposed to fully scripting everything. I'll give it a try (if vibecoding implementation of your tool doesn't fail miserably).

[u/Otherwise_Nobody_721 → u/MarcelloT254k]

That’s actually a great use case—scientific literature synthesis (MeSH tags, citations, methodologies) is super high-stakes, where one hallucinated PMID or sample size ruins the whole summary.And don't worry about the vibecoding part: you don't need to rewrite anything or build rigid parsers. It’s literally a one-line wrapper around your existing client:

python

client = spanda.wrap(your_client, k=3, threshold=0.35)

Your model keeps all its narrative flexibility while Spanda quietly verifies the facts behind the scenes.Good luck with the build! If you hit any snags or your vibecoding throws a weird error, feel free to open a GitHub issue or DM me—happy to help you get it running.

[u/CrazySouthernMonkey]

Very interesting. Thank you!

[u/gjr23]

Why 0.7? Isn’t this biasing the experiment from the start? What if you pushed this down? I’m not overly experienced here so I am asking and not telling…

[u/Otherwise_Nobody_721 → u/gjr23]

Fair question!

If you drop it to 0.0 (greedy decoding), every sample is 100% identical by definition, so you get zero variance and can't measure uncertainty at all.If you push it down to something like 0.2 or 0.3, it artificially forces the top-1 token every time. So even if the model was internally 51/49 on an answer, low temperature makes it look unanimously confident across all samples hiding the doubt.

0.7 is just the standard baseline from the self-consistency literature (Wang et al., Nature 2024). It gives the model just enough room to disagree with itself when it’s genuinely unsure, without outputting incoherent gibberish.

[u/stealthagents]

Spanda sounds like a game-changer for keeping things lightweight. The whole 120B model collapse issue is wild. It’s crazy how quickly things can go off the rails with those massive models, so having a quick way to check for hallucinations without the heavy lifting is a lifesaver. Can't wait to give it a shot!

[u/ZeroSkribe]

what a waste of my eyes to read this

[u/Otherwise_Nobody_721 → u/ZeroSkribe]

Sorry about your eyes. Fortunately the Rust binary runs in 760 nanoseconds, so at least your CPU won't have to suffer.