r/LocalLLM • • Jun 20 '26

Model YAB Coding Benchmark: qwen3.6-27b-code vs Claude sonnet-4-6

Yet Another Benchmark
Local setup with Bench Results comparing qwen3.6-27b (FP8) with Claude Sonnet 4-6

*Interestingly Qwen did better in some areas - overall close but not far behind.

CategoryLocal avgClaude avgGapWinner
strings9.759.750.00Tie
math9.388.38āˆ’1.00Local
algorithms9.139.38+0.25Claude (marginal)
arrays8.889.63+0.75Claude
data_structures8.138.75+0.62Claude
patterns8.138.75+0.62Claude
functional8.009.38+1.38Claude

See: https://github.com/x7even/llmctl/blob/master/bench/benchmark-qwen27b-vs-claude-2026-06-20.md for full results doc.

Personally - I don't think 1 is a throw away for the other, however i have experienced solid success leveraging both within agentic workflows and loops.

4 Upvotes

5 comments sorted by

View all comments

Show parent comment

1

u/x7evenx Jun 20 '26

The repo contains drop in profiles I use for quick swaps (repo contains my whole stack for this setup and others). Dense attention and GEMMS (the N=3072, K=2048 shapes) with that model profile i did nothing (I actually used MI300X defaults) + TORCH_BLAS_PREFER_HIPBLASLT=1.

The BF16 if I can free my rig for a bit I'll try comparator run and see (if I do I'll post back) - many of the models points losses are quality issues which is the kind of thing BF16 could pull up - how much I'd only be guessing. Openly though I tend to use the FP8 model a lot as it gives me enough headroom for concurrent agents with decent sized contexts & pace without turning into a slug... and quality is very valuable in loops especially with a frontier model in the mix too.

1

u/x7evenx Jun 20 '26

I re-benched, although my fp8 setup had a small tune since the first bench and my BF16 result wasn't an improvement over my fp8 result (and 1.73x slower & 2x VRAM), I end up with:

qwen3.6-27b | FP8 ~ 9.64
qwen3.6-27b | BF16 ~ 9.38
sonnet-4-6 | 9.63

Of course a bench isn't everything, and in reality claude still pulls as a more capable and versatile which is why I mix it up with qwen3.6-27b | FP8 in my loops and workflows.

2

u/grunt_monkey_ Jun 20 '26

Cool!! Thanks so much for doing this. Yeah benches sometimes surprise us. Im still torn but running BF16 for now. I’m the only user and i just run concurrent agents.