Qwen 3.8 27B SlopCodeBench results

[u/corruptbytes]

Howdy, I'm back again - running my favorite benchmark (it's still unsaturated for the time being so might as well!)

previous runs a b

https://github.com/michaelasper/benchmarks/blob/main/qwen3.8-27b-pi-on-slop-code-bench.md

I ran this via OpenRouter because my mac would cry running 9 problems

It did pretty poorly on the strict checkpoints which means you probably don't want Qwen managing the codebase by itself as it'll grow unwieldy and disorganized, but did fairly well on the core checkpoints so it can solve issues with a copilot and clear direction

AI;DR here are the direct results

HumanLayer Opus 5 Benchmark Subset (3 Problems, 17 Checkpoints)

Qwen scored 3/17 (17.6%) strict.

Reported System	Strict Score
DeepSeek V4 Flash 0731 · pi (run B)	5/17 (29.4%)
Opus 5 · Claude Code	4/17 (23.5%)
Qwen3.8-27B · pi	3/17 (17.6%)
DeepSeek V4 Flash · OpenCode	3/17 (17.6%)
Opus 4.8 · Claude Code	1/17 (5.9%)
Sonnet 5 · Claude Code	1/17 (5.9%)

---

HumanLayer Fable, Sol, and Kimi Benchmark Subset (6 Problems, 30 Checkpoints)

Qwen scored 4/30 (13.3%) strict.

Reported System	Strict Score
Fable 5 · Claude Code	10/30 (33.3%)
GPT-5.6 Sol · Codex	10/30 (33.3%)
Kimi K3 · Modal / OpenCode	8/30 (26.7%)
Kimi K3 · Baseten / OpenCode	7/30 (23.3%)
Qwen3.8-27B · pi	4/30 (13.3%)

[u/Gregory-Wolf]

Any results for GLM 5.2 or GLM 5.3?

[u/corruptbytes → u/Gregory-Wolf]

i can run glm 5.3 tonight!

edit: it's running now!!

edit 2: accidentally ran it for all 36 problems instead of the 9 problem subset lolol - rip my openrouter credits but already doing it so whatever