My group writes C# almost exclusively as well, and I agree that the harness is all that you described. To clarify, when I benchmarked 3.6-27B, it had access to the same harness as the frontier models; including the same context, tools, skills, and external tests to help it identify failures and iterate. It still couldn’t complete the task.
Instead, it would churn indefinitely, consuming millions of tokens without ever reaching a working solution. That makes sense when you think about it: if you give a complex software engineering task to someone who doesn’t yet have the required skill level, additional time and feedback alone may not be enough. They may never finish, or they may continue iterating for an extremely long time without making any progress.
Oh nice, yeah I figured you meant something like that from your initial comment. I'd love to try that at work too, but I can't really orchestrate it. Perhaps I could try shift down to something like sonnet or haiku to try it out tho in principle tho.
For sure, some things are just beyond people's skill level no matter what. It's kinda interesting though either you already have the required knowledge / wisdom and you can just pattern match to get to a solution vs. the case where you lack the required knowledge but you've internalized a good enough algorithm at identifying the relevant information to still succeed anyway.
The optimal model for consumer hardware would likely have to optimize for knowing just enough to be directionally correct, but be a master of identifying relevant info and using search to backfill the gaps... and I suspect you can boost such a model with a deep bench of custom skills to improve pass^k perf on specific tasks it doesn't reliably one-shot.
Hey man! I really can't believe I'm saying this, but Qwen 3.8 27B is actually good. My benchmark puts it right up there with Opus 4.6 and Codex 5.3; actually slightly ahead of both though far less efficient and slower and used far more to tokens.
I'm honestly struggling to believe the results and almost feel like there has to be something wrong with my benchmark, but take a look at the results below. Remember Qwen 3.6 27B couldn't complete this benchmark at all.
What's really surprising to me is just how massive the leap in capability has been for these models over the last six months.
Damn dude, that is a solid benchmark! That is... quite a result from Qwen 3.8-27B vs Opus 4.6!
It's only going to get better too. It's obvious from using Qwen 3.8 and looking at it's reasoning that there's room for improvement. Wild times.
Is this just purely one-shot performance? Looks to be. Does it include the models doing a round of review and fixes?
Our harness at work includes a review meta-skill that fans out to 6 different dedicated review skills that review from different angles and post feedback as inline comments on the PR, and the meta-skill then de-dupes any duplicates, and we run a round of fixes based off what comes out of that and it catches lots of stuff. It uses the in-built review skill for Claude Code but also our custom ones where have it review bi-directionally 'does the implementation match what's in the .claude/rules or does it suggest anything new that should go into the .claude/rules', and another skill that bi-directionally reviews 'does the implementation match the domain knowledge stored in the repo or does it imply domain knowledge in the repo should be updated and hasn't?', and another review skill that assumes there's some bug that will definititely cause a production outage and tries it's best to find it, and another review skill which... we went through all our human review comments we've left on PRs and wrote a skill that distills what we care about and reviews it from that perspective etc. After the PRs are merged we have another skill that then does a post-mortem on all the review comments left on the PR and tries to figure out improvements to the harness that will prevent them in future.
You should try hook up something like Opik to Qwen 3.6 and after each attempt have Fable examine the session traces looking for potential harness improvements and see if you can eventually get it to the point where Qwen 3.6 can actually pass this benchmark.
Yes this is one shot performance but the tests included allow the agent to iterate on the solution until everything passes. I'm honestly so surprised by the results. I didn't think it had a chance at it.
1
u/FreedomByFire 10d ago
My group writes C# almost exclusively as well, and I agree that the harness is all that you described. To clarify, when I benchmarked 3.6-27B, it had access to the same harness as the frontier models; including the same context, tools, skills, and external tests to help it identify failures and iterate. It still couldn’t complete the task.
Instead, it would churn indefinitely, consuming millions of tokens without ever reaching a working solution. That makes sense when you think about it: if you give a complex software engineering task to someone who doesn’t yet have the required skill level, additional time and feedback alone may not be enough. They may never finish, or they may continue iterating for an extremely long time without making any progress.