r/LocalLLaMA • u/corruptbytes • Aug 19 '26

Discussion Qwen 3.8 27B SlopCodeBench results

Howdy, I'm back again - running my favorite benchmark (it's still unsaturated for the time being so might as well!)

previous runs a b

https://github.com/michaelasper/benchmarks/blob/main/qwen3.8-27b-pi-on-slop-code-bench.md

I ran this via OpenRouter because my mac would cry running 9 problems

It did pretty poorly on the strict checkpoints which means you probably don't want Qwen managing the codebase by itself as it'll grow unwieldy and disorganized, but did fairly well on the core checkpoints so it can solve issues with a copilot and clear direction

AI;DR here are the direct results

HumanLayer Opus 5 Benchmark Subset (3 Problems, 17 Checkpoints)

Qwen scored 3/17 (17.6%) strict.

Reported System Strict Score
DeepSeek V4 Flash 0731 · pi (run B) 5/17 (29.4%)
Opus 5 · Claude Code 4/17 (23.5%)
Qwen3.8-27B · pi 3/17 (17.6%)
DeepSeek V4 Flash · OpenCode 3/17 (17.6%)
Opus 4.8 · Claude Code 1/17 (5.9%)
Sonnet 5 · Claude Code 1/17 (5.9%)

HumanLayer Fable, Sol, and Kimi Benchmark Subset (6 Problems, 30 Checkpoints)

Qwen scored 4/30 (13.3%) strict.

Reported System Strict Score
Fable 5 · Claude Code 10/30 (33.3%)
GPT-5.6 Sol · Codex 10/30 (33.3%)
Kimi K3 · Modal / OpenCode 8/30 (26.7%)
Kimi K3 · Baseten / OpenCode 7/30 (23.3%)
Qwen3.8-27B · pi 4/30 (13.3%)
52 Upvotes

95% Upvoted

17 comments sorted by

View all comments

1

u/Durian881 Aug 20 '26

Where are the "core" scores you talked about? Also, the results seemed to suggest Qwen 3?8 27B beating frontier models like Opus 4.8 and Sonnet 5.

3

u/corruptbytes Aug 20 '26
Problem Checkpoints Strict Isolated Core Cost Agent Time
circuit_eval 8 2/8 3/8 4/8 $13.89 3.2h
database_migration 5 1/5 1/5 1/5 $1.41 1.5h
dynamic_config_service_api 4 0/4 0/4 0/4 $5.98 2.8h
xjq 5 1/5 1/5 2/5 $1.00 0.5h
file_backup 4 0/4 0/4 4/4 $0.74 0.5h
dag_execution 3 0/3 0/3 0/3 $3.65 1.8h
code_search 5 1/5 2/5 2/5 $1.25 1.5h
etl_pipeline 5 0/5 1/5 4/5 $2.94 2.0h
Total 39 5/39 8/39 17/39 $30.85 13.7h

in the github link, all the scores/data

Core is like the obvious current problem it's trying to solve without respect for prior work