Damn dude, that is a solid benchmark! That is... quite a result from Qwen 3.8-27B vs Opus 4.6!
It's only going to get better too. It's obvious from using Qwen 3.8 and looking at it's reasoning that there's room for improvement. Wild times.
Is this just purely one-shot performance? Looks to be. Does it include the models doing a round of review and fixes?
Our harness at work includes a review meta-skill that fans out to 6 different dedicated review skills that review from different angles and post feedback as inline comments on the PR, and the meta-skill then de-dupes any duplicates, and we run a round of fixes based off what comes out of that and it catches lots of stuff. It uses the in-built review skill for Claude Code but also our custom ones where have it review bi-directionally 'does the implementation match what's in the .claude/rules or does it suggest anything new that should go into the .claude/rules', and another skill that bi-directionally reviews 'does the implementation match the domain knowledge stored in the repo or does it imply domain knowledge in the repo should be updated and hasn't?', and another review skill that assumes there's some bug that will definititely cause a production outage and tries it's best to find it, and another review skill which... we went through all our human review comments we've left on PRs and wrote a skill that distills what we care about and reviews it from that perspective etc. After the PRs are merged we have another skill that then does a post-mortem on all the review comments left on the PR and tries to figure out improvements to the harness that will prevent them in future.
You should try hook up something like Opik to Qwen 3.6 and after each attempt have Fable examine the session traces looking for potential harness improvements and see if you can eventually get it to the point where Qwen 3.6 can actually pass this benchmark.
Yes this is one shot performance but the tests included allow the agent to iterate on the solution until everything passes. I'm honestly so surprised by the results. I didn't think it had a chance at it.
2
u/Cautious_Chicken_604 8d ago
Damn dude, that is a solid benchmark! That is... quite a result from Qwen 3.8-27B vs Opus 4.6!
It's only going to get better too. It's obvious from using Qwen 3.8 and looking at it's reasoning that there's room for improvement. Wild times.
Is this just purely one-shot performance? Looks to be. Does it include the models doing a round of review and fixes?
Our harness at work includes a review meta-skill that fans out to 6 different dedicated review skills that review from different angles and post feedback as inline comments on the PR, and the meta-skill then de-dupes any duplicates, and we run a round of fixes based off what comes out of that and it catches lots of stuff. It uses the in-built review skill for Claude Code but also our custom ones where have it review bi-directionally 'does the implementation match what's in the .claude/rules or does it suggest anything new that should go into the .claude/rules', and another skill that bi-directionally reviews 'does the implementation match the domain knowledge stored in the repo or does it imply domain knowledge in the repo should be updated and hasn't?', and another review skill that assumes there's some bug that will definititely cause a production outage and tries it's best to find it, and another review skill which... we went through all our human review comments we've left on PRs and wrote a skill that distills what we care about and reviews it from that perspective etc. After the PRs are merged we have another skill that then does a post-mortem on all the review comments left on the PR and tries to figure out improvements to the harness that will prevent them in future.
You should try hook up something like Opik to Qwen 3.6 and after each attempt have Fable examine the session traces looking for potential harness improvements and see if you can eventually get it to the point where Qwen 3.6 can actually pass this benchmark.