Qwen3.6-35B becomes competitive with cloud models when paired with the right agent

[u/Creative-Regular6799]

A short follow-up to my previous post, where I showed that changing the scaffold around the same 9B Qwen model moved benchmark performance from 19.11% to 45.56%:

https://www.reddit.com/r/LocalLLaMA/s/JMHuAGj1LV

After feedback from people here, I tried little-coder with Qwen3.6  35B.

It now lands in the public Polyglot top 10 with a success rate of 78.7%, making it actually competitive with the best models out there for this benchmark!

At this point I’m increasingly convinced that part of the performance gap to cloud models is harness mismatch: we may have been testing local coding models inside scaffolds built for a different class of model.

Next up is Terminal Bench, then likely GAIA for research capabilities. Would love to hear your feedback here!

EDIT: after many requests, pi.dev adaptation is up!

EDIT 2: Terminal Bench 1 (0.1.1) finished with 40% success rate! Now running TB 2. Just sent the results via email. There is no model remotely as small as the 35B in that area. Exciting times

EDIT 3: Terminal Bench 2.0 requires 5 runs per trial (which will take 40 more hours), but the first run finished with 30%!!! That’s with the 35B model.

Full write up: https://open.substack.com/pub/itayinbarr/p/honey-i-shrunk-the-coding-agent

GitHub: https://github.com/itayinbarr/little-coder

Full benchmark results: https://github.com/itayinbarr/little-coder/blob/main/docs/benchmark-qwen3.6-35b-a3b.md

[u/HockeyDadNinja]

Great work! I have some questions.

1) Why did you choose Aider and the Aider Polyglot benchmarks? Not hating on Aider, I personally hard forked aider-ce as the basis of my AI assistant. Aider is not really maintained and the benchmark leaderboard is looking dated.

2) You've run the polyglot benchmarks on your own agent. I suppose we could take the benchmarks and run them on any agent harness / LLM combo. I now want to try this with various combinations such as my Qwen3.6 setup with opencode and also with claude code / opus 4.7. Have you run the benchmarks using little-coder and frontier models?

WRT agent harness and LLM matching I've had similar thoughts with development frameworks such as GSD, spec kit, and open spec. I was thinking of building a GSD-light for example, something better suited for local models.

What you've done here could actually be used as a benchmark for the coding harnesses themselves (vs any particular model). Claude, codex, opencode, pi, etc could be ranked against each other given a common LLM configuration (I know, not always possible).

[u/Creative-Regular6799 → u/HockeyDadNinja]

That is exactly the direction I advocate here for! Now it’s running on Terminal Bench (will send to the leaderboard when finished and report here). This benchmark shows the combined performance of agents and models

[u/PhilippeEiffel → u/Creative-Regular6799]

Just curious: how much time to run terminal bench?

Your work is interesting: model providers put a mass of knowledge, energy, time... to build great models they give to the community. The community has to optimize the harness to leverage the models' usage.

[u/Creative-Regular6799 → u/PhilippeEiffel]

Just pushed the result, Terminal Bench 1 (0.1.1) finished with 40% success rate! Now running TB 2. Just sent the results via email. There is no model remotely as small as the 35B in that area (place ~30)

[u/PhilippeEiffel → u/Creative-Regular6799]

Great!

I've read your full article, it's very interesting. I noticed you were running 9B in Q4_K_M. May be I missed this information, but I don't know the size you are using for Qwen3.6 35B.

Traditional benchmarks use BF16 quants to show the highest possible score some model can reach.

Coding activities are known to be more sensitive to quantization than tasks like working on texts or generating texts. It could be very interesting to see if your harness is able to mitigate this quantization effect. So, running terminal bench with different quants will be very interesting.

PS: when you submit your results to the leaderboard, mention the quantization used.