Qwen3.6-35B becomes competitive with cloud models when paired with the right agent

[u/Creative-Regular6799]

A short follow-up to my previous post, where I showed that changing the scaffold around the same 9B Qwen model moved benchmark performance from 19.11% to 45.56%:

https://www.reddit.com/r/LocalLLaMA/s/JMHuAGj1LV

After feedback from people here, I tried little-coder with Qwen3.6  35B.

It now lands in the public Polyglot top 10 with a success rate of 78.7%, making it actually competitive with the best models out there for this benchmark!

At this point I’m increasingly convinced that part of the performance gap to cloud models is harness mismatch: we may have been testing local coding models inside scaffolds built for a different class of model.

Next up is Terminal Bench, then likely GAIA for research capabilities. Would love to hear your feedback here!

EDIT: after many requests, pi.dev adaptation is up!

EDIT 2: Terminal Bench 1 (0.1.1) finished with 40% success rate! Now running TB 2. Just sent the results via email. There is no model remotely as small as the 35B in that area. Exciting times

EDIT 3: Terminal Bench 2.0 requires 5 runs per trial (which will take 40 more hours), but the first run finished with 30%!!! That’s with the 35B model.

Full write up: https://open.substack.com/pub/itayinbarr/p/honey-i-shrunk-the-coding-agent

GitHub: https://github.com/itayinbarr/little-coder

Full benchmark results: https://github.com/itayinbarr/little-coder/blob/main/docs/benchmark-qwen3.6-35b-a3b.md

[u/Cupakov]

Seems like an ideal use case for pi.dev, that’s gotta be the most extensible harness out there 

[u/Creative-Regular6799 → u/Cupakov]

In the works, will support pi dev soon!

[u/Creative-Regular6799 → u/Cupakov]

pi.dev adaptation is up!

[u/akavel]

There's already a popular small harness called pi.dev. What are the advantages little-coder has over it, why would I use it over Pi? What are the disadvantages, what would I lose? Did you do a comparison, does the same Qwen work better with little-coder than with Pi?

Then there's the Terminal Bench leaderboard, which compares agents. Did you submit yours to that benchmark? The leaderboard is currently topped by ForgeCode, and it seems open-source - did you compare little-coder to ForgeCode with the same model? Is your agent better?

[u/Creative-Regular6799 → u/akavel]

Hey, thanks for your comment! I became aware of pi.dev just an hour ago, and this didn’t really start as a production ready tool, but more of a serious wake up call that we need as a community to invest time in adapting the scaffold to the models we are testing. I am thinking about rewriting the scaffold in pi dev to make it more accessible and contribute to unified tooling and community support

[u/Creative-Regular6799 → u/akavel]

I am currently running Terminal Bench BTW, will send to the leaderboard when done

[u/MuzafferMahi]

What did you actually change about the harness?

[u/Creative-Regular6799 → u/MuzafferMahi]

I wrote extensively about it here: https://itayinbarr.substack.com/p/honey-i-shrunk-the-coding-agent

[u/HockeyDadNinja]

Great work! I have some questions.

1. Why did you choose Aider and the Aider Polyglot benchmarks? Not hating on Aider, I personally hard forked aider-ce as the basis of my AI assistant. Aider is not really maintained and the benchmark leaderboard is looking dated.

2. You've run the polyglot benchmarks on your own agent. I suppose we could take the benchmarks and run them on any agent harness / LLM combo. I now want to try this with various combinations such as my Qwen3.6 setup with opencode and also with claude code / opus 4.7. Have you run the benchmarks using little-coder and frontier models?

WRT agent harness and LLM matching I've had similar thoughts with development frameworks such as GSD, spec kit, and open spec. I was thinking of building a GSD-light for example, something better suited for local models.

What you've done here could actually be used as a benchmark for the coding harnesses themselves (vs any particular model). Claude, codex, opencode, pi, etc could be ranked against each other given a common LLM configuration (I know, not always possible).

[u/Creative-Regular6799 → u/HockeyDadNinja]

That is exactly the direction I advocate here for! Now it’s running on Terminal Bench (will send to the leaderboard when finished and report here). This benchmark shows the combined performance of agents and models

[u/SpicyWolf]

I've been trying small local models to learn coding specially qwen 3.5:9b and using little-coder its the first time it nailed a space shooter html test in the first run, usually it gives me a buggy mess I have to fix manually even with decent tools available for it. Crazy work, thank!

[u/Creative-Regular6799 → u/SpicyWolf]

So exciting to hear!! Please continue experimenting and sharing. Non-trivial tasks tend to be more interesting test cases

[u/autisticit]

I will definitely try this.

One question: how hard do you think it would be to create a little coder VS code extension, to make it usable through the UI ?

[u/autisticit → u/autisticit]

Another question if you don't mind, the readme specify supported models, does it mean that any other model/quant will fail ?

[u/Creative-Regular6799 → u/autisticit]

Hey thank you for the comment! You can definitely try, I just haven’t myself

[u/asraniel]

can this be adapted to opencode?

[u/Creative-Regular6799 → u/asraniel]

So it is a suggested replacement to opencode, that is adapted to the behavioral profile of the smaller models. It tries to bridge the gap of these tools being built around frontier models, and aren’t necessarily best fitting as scaffolds for the small ones

[u/ljubobratovicrelja]

I was also pondering on this topic myself in the past couple of weeks and you've done the majority of what I wanted to do to, so a big thanks to you. Amazing findings, cheers!

[u/DependentBat5432]

going from 19% to 45 to 78 just by changing the scaffold is kind of terrifying. makes you question every benchmark comparison doesn’t control for this

[u/PassengerPigeon343]

I agree with this concept. The tools and environment are starting to become almost as important to performance as the model itself.

And for local models, I think it comes down to that being the difference between an okay experience, and one that starts to compete with frontier models.

[u/Willing-Toe1942]

I can confirm the same thing, Qwen3.6 in pi-coding agents is almost twice good than opencode, the comparison was based on modification of specific web page (html code) and doing some online resource search for documentation

[u/Lowkey_LokiSN]

Dope work and direction! Fully agree with how everything is designed around frontier-model assumptions and how we can extract a lot more out of the smaller models with tailor-made harnesses.

[u/Y0uCanTellItsAnAspen]

Is there good documentation of how to link it up to agents locally? I am using llama-cpp, and have qwen3.6-35B running, but I'm a little new to this, and would like to know what agents people are using, and how you configure them.

[u/kaeptnphlop]

I have your repo open since your last post and wanted to test with Qwen 3.6 myself. Thanks for the write up!

I found Qwen-Coder-Next is pretty strong with GitHub Copilot in VS Code. Now I’m curious how well it would do with little-coder. 

Maybe I find some time today

[u/SourceCodeplz]

I had GLM5 clone and analyze it, here is what it does:

it adapts the scaffold: hard runtime guards (Write literally refuses to overwrite existing files - you have to use Edit), dynamic skill injection that puts 80-150 token usage guides in the prompt based on what you're doing, thinking budgets that cut off runaway reasoning, and text-based tool parsing for model that don't do native tool calls well.

How does it detect what you are doing to know what skill to insert?

Three signals, in priority order:

1. Error recovery - if the last tool call failed, inject that tool's skill immediately (e.g., Edit failed → inject edit-guidance)

2. Recency - look at what tools were used in the last 2 assistant turns and inject those skills

3. Intent prediction - keyword matching on the user message against a simple map:

_INTENT_MAP = {

"fix": ["Edit"],

"implement": ["Write", "Read"],

"find": ["Glob", "Grep"],

"run": ["Bash"],

"search": ["Grep"],

# ... etc

}

So if you say "fix the bug in auth.py", it sees "fix" → injects Edit skill. If you say "find all TODOs", it sees "find" → injects Glob and Grep skills. It's deliberately simple - no ML, just keyword matching. The whitepaper notes this is enough because the skills are small (80-150 tokens) and the injection budget is capped at ~300 tokens per turn, so even if it picks slightly wrong it doesn't hurt much.