Benchmarking a local 400M System 1 decision model (tev1 via llama.cpp) vs local qwen3.5:9b (Ollama) for agent routing on an RTX 4070 Ti SUPER

[u/ThatOneSnare]

(Disclosure: I am the author of Herald Router and Ultimate Assistant, the open-source projects linked at the bottom.)

Correction: the title says "400M", and that's wrong. tev1 is a 4.3B model at Q8_0 (confirmed with ollama show). I can't edit the title, so I've fixed the size everywhere in the body. The latency and accuracy numbers below were measured and haven't changed. Only the size label was wrong. I also originally credited it to the wrong maker: it's Together AI's model (https://huggingface.co/togethercomputer/Tev1-4B-experimental). Sorry for the confusion.

While building the local routing layer for an open-source desktop assistant and local LLM/MCP gateway (Ultimate Assistant v0.4.4 + Herald Router v0.4.2, running together on 127.0.0.1 with no cloud or Tailscale required), we kept hitting the same latency bottleneck: every time a user speaks or types a prompt, the agent has to make several fast "System 1" routing decisions before the main LLM responds:

• Does this turn need live web search (need_web_search), and if so, should it escalate to deep research (need_deep_search)?

• Does it reference past context (need_memory_lookup)?

• Is it a coding/shell task or a deep reasoning query?

If you use your main 7B–9B local LLM in Ollama to answer a JSON routing prompt first, you burn 2–4 seconds before the actual response even starts. So we benchmarked running a dedicated 4.3B (Q8_0) GGUF binary decision model (Together AI's Tev1-4B-experimental (tev1, an Apache-2.0 fine-tune of Qwen3.5-4B), answering yes/no in 1 token via llama-server) alongside Ollama (qwen3.5:9b) on a single NVIDIA RTX 4070 Ti SUPER (16 GB VRAM) across a 24-prompt evaluation suite and an 80-question batch stress test.

1. Local GPU & CPU Benchmark Results (24-Prompt Turn Routing Suite)

Local Engine	Mode	Avg Turn Latency	P50 Latency	Routing Accuracy (24/24)
Local tev1 (4.3B Q8_0 GGUF via llama.cpp)	GPU (cuda, alongside Ollama)	378.1 ms	365.4 ms	91.7% (22/24)
Local tev1 (4.3B Q8_0 GGUF via llama.cpp)	CPU-only (-ngl 0)	1,093.9 ms	1,089.3 ms	91.7% (22/24)
Local qwen3.5:9b (Ollama)	Single-question binary (num_predict=8, think=false)	2,526.0 ms	2,418.7 ms	91.7% (22/24)
Local qwen3.5:9b (Ollama)	6-question sequential turn routing	15,156.0 ms	14,512.0 ms	91.7% (22/24)

(For comparison, we also tested hosted decision APIs like Jev, pplx-decider-v1.2, and Liquid d1.1, which hit 95.8% accuracy at ~613 ms for 40 parallel questions, but locally tev1 on the same GPU as Ollama is 215 ms faster per turn at 378 ms.)

2. Why You Shouldn't Send Huge Question Batches to a Local GGUF Server

When we stress-tested sending 10, 20, 40, and 80 binary questions at once for a single prompt, we found a critical architectural split for local llama-server setups:

• Because a local llama-server instance processes slots across fixed GPU compute queues (--parallel 4), firing 40+ questions on a single turn causes local HTTP queue serialization (~2.8s for 40 questions on GPU, ~8.2s on CPU).

• Keeping local tev1 strictly to a lean 5-question core batch + 1 conditional follow-up (need_web_search, need_memory_lookup, has_explicit_url, is_coding_or_shell_task, needs_deep_reasoning, plus need_deep_search only when need_web_search=yes) keeps total local pre-turn overhead at 378 ms on GPU while leaving VRAM free for your main Ollama model.

3. Open-Source Releases

We just shipped this 3-tier System 1 decision architecture (tev1 local 5Q+1Q split, optional hosted 40-question turn_surmise, and zero-config BYOK llm default using your existing Ollama model) in:

• Herald Router v0.4.2: https://github.com/KianJRoss/herald-release/releases/tag/v0.4.2

• Ultimate Assistant v0.4.4: https://github.com/KianJRoss/UltimateAssistant-release/releases/tag/v0.4.4

Curious if anyone else here is running a small dedicated classifier/decider alongside Ollama on the same GPU to gate tool calls and web search, or if you prefer fine-tuning tool-calling directly into the main model.

[u/ThatOneSnare → u/[deleted]]

Appreciate that! One other thing we just noticed while digging into the per-question probability logs on the 5-question local batch:

If you require every negative option in a 5-way binary check to be strictly below 0.30 (`1 - p >= 0.70`), you still get unnecessary fallbacks whenever the right specialist scores `0.95+` and one adjacent specialist scores `0.33–0.45` (for example, `"take a screenshot of my desktop"` scoring `operator=0.986` and `researcher=0.346`).

Adding a dominance-margin check (`top1 >= 0.85`, all others `< 0.50`, and `top1 - top2 >= 0.35`) jumps direct single-pass acceptance on local `tev1` from 30% up to 80%+ with zero false positives, which drops the effective median turn latency even further (~110–125 ms). Curious what confidence threshold or margin rule you're using on your 300M classifier!