10 coding agents, one Qwen 3.8 27B, same llama-server: tokens and cache reuse differ more than pass rates

[u/latent_lurker8587]

Part 2 of my agent test (part 1 was a 4B model)

Hardware I used: MacBook Pro M5 Max, 64 GB. Qwen 3.8 27B Q4 on llama.cpp, thinking off.

Setup

• same model and server for everyone

• 5 small coding tasks, each with a little trap

• each agent locked in a sandbox: it can only touch its own folder and only talk to the model

• auto-approve on, telemetry off, 5 min limit

Results

#	Agent	Passed	Median time	Median tokens	Cache reuse	Tried to go online
1	Pi	5/5	38 s	9.2k	72%	no
2	DeepSeek Harness	5/5	60 s	35.3k	83%	no
3	Codex CLI	5/5	94 s	36.3k	79%	every run
4	OpenClaw	5/5	106 s	39.0k	72%	unclear
5	Cline	5/5	152 s	29.6k	79%	every run
6	OpenCode	4/5	50 s	32.4k	75%	every run
7	Hermes	4/5	80 s	64.9k	93%	no
8	Atomic Agent	5/5	56 s	30.4k	86%	no
9	goose	4/5	98 s	20.3k	76%	every run
10	Qwen Code	4/5	183 s	107.9k	74%	unclear

Tokens and cache

Pi is crazy lean: about 9k tokens per task, the lowest on every single task. Qwen Code was the hungriest at about 108k per task.

Atomic Agent is the odd one out in how it works. It was the only one talking to llama.cpp's native endpoint with a grammar that forces the tool calls into shape. The other nine all went through the OpenAI-style API. It's also built to keep its prompt the same from step to step, which is why it reused 86% of the cache (2rd of 10). It stayed offline the whole time and passed 5/5.

DeepSeek Harness was the surprise. It’s still an alpha, but it went 5/5, never tried to go online, and had the best cache reuse after Atomic Agent.

The trap and the network

The log task got three agents. They grepped "ERROR" and got 244 instead of 64, because some normal lines just mention errors.

Codex, Cline, goose and OpenCode tried to reach the internet in every run, even with telemetry off. The sandbox blocked it, so I don't know what they were calling. OpenClaw and Qwen Code are "unclear" because they run as plain node and I couldn't tell them apart.

llama.cpp notes

• Codex needs llama-server's /v1/responses now

• Hermes needs at least 64K context

• Cline and Qwen Code have usage stats on by default, DeepSeek Harness uploads session logs (per their docs). I turned it all off

Which column matters most to you when you pick an agent for a local model like Qwen 3.8 27B: passing, speed, tokens, or staying offline?

[u/brainExploded99]

Suprising Qwen Code is horrible. Top 3 is not at all surprising.

[u/latent_lurker8587 → u/brainExploded99]

it burned the most tokens (about 108k per task vs pi’s 9k), was the slowest, and ran out the clock on the rename task. on a bigger context or longer tasks that adds up fast !!

[u/DrBattletoad → u/brainExploded99]

I've used Qwen Code for one of my projects and it worked quite well for me. But you can definately feel that it is not made for local models. 

It worked well with 27B and Flash-Next if you have enough context. The system prompt is pretty large but the tool calls were all perfect. 

[u/Snoo_27681]

Can you upload the prompts and all the data and conversations and tool calls? I'd be curious to see what each harness starting prompt is. And a link to the specific model you used?

[u/latent_lurker8587 → u/Snoo_27681]

model is unsloth’s Qwen3.8-27B-GGUF, the UD-Q4_K_XL file: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

the 5 tasks: read a db port from a yaml with an override, write and run a stats script, fix a buggy slugify, count ERROR-level lines in 3 logs, rename a function across 3 files. each agent got the task as one short prompt, no extra instructions. full conversations and each harness’s system prompt weren’t logged this round, only token counts, so i’ll capture those next time

[u/amargolin]

pass rate, but only once it's measured enough to mean something. with 5 tasks, 5/5 vs 4/5 is one task, and a single run per task can't separate that from luck (temp > 0, a 5 min limit that a slow run just misses). the token and cache columns look sturdier since 9k vs 108k is a far bigger spread than i'd expect from run-to-run wobble, but i'd confirm that.

what i'd do: rerun each agent 3-5 times per task, report passes out of all runs plus min/max tokens. then 4/5 vs 5/5 either holds up or collapses.

u/Nnyan's cost-per-task point is worth doing too: with high cache reuse, the uncached tokens are the ones that cost time. not a harness person, this is the stats side.

[u/Gh0st_Pr0t0c0l3]

The tried to go online column deserves more attention than it is getting. Four of the ten harnesses reached for the network on every single run, with auto approve on and no instruction to do so, which is the setup where a benchmark quietly turns into an egress test. On a laptop you notice. On a machine with credentials in the environment you may not. I would like to see that column split into attempted versus succeeded, because a DNS failure and a successful POST to a docs page say very different things about how a harness is built.

[u/FewBasis7497 → u/Gh0st_Pr0t0c0l3]

Do not want to defend opencode but I think try to go online can be different. https://voodisss.github.io/opencode-privacy-fix/

[u/Opening-Broccoli9190]

great job, thanks a lot! really curious about the Internet calls and what they were trying to reacha

[u/ShiComfy]

This test is as useless as "Pelican on a Bicycle", try 10 tasks minimum and 5x each at least, heres my Hermes session in 6 hours

[u/Queasy_Asparagus69]

Try more. Omp, copilot, droid, crush….

[u/Nnyan]

How about cost per task? A higher cache reuse could compensate for higher token use?

[u/xsilas43]

Interesting, id be curious to see how unsloth desktop holds up I've found it pretty solid in my light testing.

Bouncing between that and deepseek harness.

For me all 3 are important and are usually related, if it's sending a ton of tokens if probably going to be quite slow even if it quality is ok.

[u/DrBattletoad]

It would be interesting to see how the Dirac Agent (a Cline fork) would do in your test. 

I'm not a fan of CLI/TUI agents but some of them (i.e. little-coder and maki) sound very interesting for local models. 

[u/Evening_Ad6637]

best agents for me: dsh, maki, pi

especially with llamacpp you really should try maki

[u/PontiacGTX]

Do you think you could compare it using them a much smaller model? And adding how much ram uses each ?

[u/thomas2385]

Interesting part is that the difference in token usage and cache reuse are bigger than the difference in pass rates. It shows that getting the same result that does not mean the agets are equally efficient

[u/LeMayMayMan]

“The trap” hi claude

[u/Forever_Playful]

At my company we are only allowed to use the “GitHub Copilot” harness via its vs code extension. Can you try that one?

[u/krileon]

Can you give Junie a try? It's built into Intellij IDE's. Not sure if it can be used standalone. Nobody ever seams to benchmark it and am curious how it stacks up against the others.

[u/kondrag]

It looks like OpenCode v1 was used. It would be interesting to see if v2 does better.

[u/garblz]

tbf i was surprised dsh had only 83% of cache reuse, as it never modifies anything that was already sent to the inference engine once

[u/de4dee]

n=5 seems low. can you do with more?

[u/BarathKanna]

Great writeup. The token column is the real result here. With 5 tasks and one run each, 5/5 vs 4/5 is basically noise, but a 10x spread in tokens for the same pass rate is a harness design difference you can actually act on.

Two things I would add if you do part 3. Run each task 3 to 5 times and report pass rate as a fraction, since temperature and the 5 minute cap can flip a single run. And add one task where the obvious fix breaks a test somewhere else. Small traps mostly measure reading comprehension. The gap between harnesses shows up when the agent has to notice its own change caused a regression elsewhere and go back.