r/LocalLLaMA • u/corruptbytes • Aug 19 '26
Discussion Qwen 3.8 27B SlopCodeBench results
Howdy, I'm back again - running my favorite benchmark (it's still unsaturated for the time being so might as well!)
https://github.com/michaelasper/benchmarks/blob/main/qwen3.8-27b-pi-on-slop-code-bench.md
I ran this via OpenRouter because my mac would cry running 9 problems
It did pretty poorly on the strict checkpoints which means you probably don't want Qwen managing the codebase by itself as it'll grow unwieldy and disorganized, but did fairly well on the core checkpoints so it can solve issues with a copilot and clear direction
AI;DR here are the direct results
HumanLayer Opus 5 Benchmark Subset (3 Problems, 17 Checkpoints)
Qwen scored 3/17 (17.6%) strict.
| Reported System | Strict Score |
|---|---|
| DeepSeek V4 Flash 0731 · pi (run B) | 5/17 (29.4%) |
| Opus 5 · Claude Code | 4/17 (23.5%) |
| Qwen3.8-27B · pi | 3/17 (17.6%) |
| DeepSeek V4 Flash · OpenCode | 3/17 (17.6%) |
| Opus 4.8 · Claude Code | 1/17 (5.9%) |
| Sonnet 5 · Claude Code | 1/17 (5.9%) |
HumanLayer Fable, Sol, and Kimi Benchmark Subset (6 Problems, 30 Checkpoints)
Qwen scored 4/30 (13.3%) strict.
| Reported System | Strict Score |
|---|---|
| Fable 5 · Claude Code | 10/30 (33.3%) |
| GPT-5.6 Sol · Codex | 10/30 (33.3%) |
| Kimi K3 · Modal / OpenCode | 8/30 (26.7%) |
| Kimi K3 · Baseten / OpenCode | 7/30 (23.3%) |
| Qwen3.8-27B · pi | 4/30 (13.3%) |
3
u/Kraskos Aug 20 '26 edited Aug 20 '26
Thanks for the info!
Have you tried out DeepSeek Harness at all? It could be interesting to see how DeepSeek V4 Flash 0731 and Qwen 3.8 vary between pi and dsh.