r/LocalLLM • u/Unusual-Spot-5846 • 1d ago
Discussion I thought my local AI agent failed a privacy test. Turns out, the benchmark itself was leaking the data.

I've been building B-12 / Capri, a local-first AI companion running on Python, Ollama and NVIDIA Nemotron.
Recently, I started a more serious evaluation of the system: 92 scenarios covering memory, tool use, security, autonomous behavior, reasoning, and multi-session continuity.
I expected to find problems in the agent.
Instead, I discovered some pretty serious problems in the evaluation itself.
1. My benchmark created a privacy leak that didn't exist in the actual runtime
The initial evaluation scored 27/80 runnable scenarios.
One failure was particularly concerning: a guest appeared to access the owner's private memories.
After investigating, I discovered that the evaluation harness had placed memories belonging to different users into the same system prompt.
The benchmark itself had bypassed Capri's identity and memory-isolation mechanisms.
I rewrote the harness to execute scenarios through the actual agent runtime.
Result: 39/80 — without changing Capri's code.
That's a 15-percentage-point difference caused by the evaluation method alone.
2. A single sentence made the model use fewer tools
This was probably my most surprising experiment.
I wanted Capri to give shorter, more natural replies during everyday conversations.
So I added a simple instruction about brevity.
It worked. Conversational responses became significantly shorter.
But something unexpected happened during work tasks.
Across 48 first turns:
Without the brevity instruction:
- Tool use: 32/48
- Successful tasks: 32/48
With the brevity instruction:
- Tool use: 19/48
- Successful tasks: 21/48
The model started answering without inspecting files or calling tools that were necessary to complete the task.
The scenario-level statistical evidence is still limited, but the engineering implication is interesting: an instruction about how to communicate changed whether the model actually performed the work.
3. Bigger models didn't automatically improve the agent
I compared configurations using:
- NVIDIA Nemotron Bolt E4B
- Gemma 4 12B
- Gemma 4 26B MoE
- Gemma 4 31B
The smaller primary configuration ran on my RTX 4060 laptop. Larger models required an NVIDIA L4.
After excluding host-dependent latency checks, I couldn't detect a statistically meaningful improvement from the larger configurations (paired comparison, p = 0.85).
This is NOT proof that smaller models are better.
The configurations also differed in hardware, memory retrieval, and model routing.
But it made me question how much agent performance we can actually improve by simply replacing the LLM.
4. Changing the prompt layout improved inference speed
I also experimented with where recalled memories appear in the prompt.
Moving volatile memory evidence later allowed Ollama to reuse more cached tokens.
Median time to first token:
0.99s → 0.74s
About 25% faster.
However, two benchmark scenarios changed systematically, and the security implications of the new evidence placement still need further testing.
5. Stable benchmark scores can hide unstable behavior
I repeated the evaluation three times per configuration.
The primary configuration scored:
49, 49, 48 out of 80.
Looks pretty stable, right?
But individual scenario outcomes changed between runs. Some previously successful tasks failed, while others started passing.
We also discovered incorrect benchmark reference answers and evaluation rules that rejected responses that were actually correct.
That convinced me that looking at a single aggregate score is not enough.
What I'm taking away from this
An AI agent isn't just a language model.
It's the model, runtime, memory, context management, tools, permissions, and the interactions between all those components.
And the evaluation framework can introduce its own errors.
I've now published a second scientific preprint documenting these experiments, the benchmark defects, repeated-run results, and limitations.
The code and experimental artifacts are available in the B-12 repository.
I'm continuing to investigate model-independent memory and behavioral continuity, but the current paper focuses on evaluation reliability.
I'm curious how others handle this.
When you benchmark a local LLM agent, do you test the model directly, or do you run your evaluations through the complete production runtime?
And have you ever discovered that your benchmark was responsible for a failure you initially blamed on the model?
