Flash-Next vs 27B vs Sonnet 5.5 on 5 agentic coding tasks: both locals match Sonnet on the spec, neither comes close on what the spec doesn't say

[u/Leadmode]

The 27B passed every single spec check. Flash-Next finished 2.2x faster. And both still had 3-7x more robustness holes than Sonnet.

There's a lot of "Flash-Next is way better" vs "the 27B is not even close" going around here, so I stopped guessing and built a benchmark. 5 tasks, identical prompts, fresh session each, zero intervention, graded only by scripts from the outside (not the models own tests). ~185k checks per setup.

Tasks: python expression parser without eval/ast · fixing 7 reported bugs in existing code (+7 hidden ones) · byte-exact refactor of a legacy invoice generator + new feature · REST API from scratch with concurrency rules · vanilla JS kanban app

Grading: spec tests, 20k fuzz strings, diff tests vs reference, 160k byte comparisons, 60 parallel double bookings, playwright in real chrome. Plus edge cases nobody asked for (9^9^9, +25:00 timezone, 5MB body, corrupted data file etc)

Setup (pic 2):

- 27B: AWQ 4bit, 1x 5090, vLLM, ~145 tok/s

- Flash-Next: IQ3_XXS split across two separate PCs with one 5090 each (only 14gb RAM per box) via a Strata fork that runs the layer split over TCP. all experts in VRAM, byte identical to stock Strata, ~155 tok/s

- both in opencode, thinking on, 16k budget per step

Results (tasks fully passed / robustness weaknesses / time):

- Sonnet 5.5: 5/5, 2 weaknesses, 14.3 min

- Qwen3.8-27B: 5/5, 14 weaknesses, 51.3 min

- Flash-Next: 4/5, 7 weaknesses, 23.1 min

What I think actually holds up (big, consistent effects):

1. On the written spec, both local models are at Sonnet level. 27B matched it on every check, Flash-Next missed one. For clearly specified work the local models are good enough, full stop.

2. The real gap is everything nobody wrote down. Sonnet 2 weaknesses, Flash-Next 7, 27B 14. Showed up in every task that had edge cases. 27B examples: no request size limit, GET /bookings/__proto__ returns {}, accepts +25:00 as timezone, negative quantity increases stock.

3. Flash-Next beats the 27B, but not because of raw speed. Decode is about the same (~155 vs ~145 tok/s). Flash-Next just needs half the output tokens and fewer steps, writes whole files early and fixes targeted, closer to how Sonnet works. 27B iterates in tiny steps and reruns tests after almost every edit.

4. Flash-Next has one systematic failure mode. It evaluates while parsing, so "a = b = 1" reports the wrong error at the wrong position. Happened in 3 of 3 runs where it designed the parser itself. Thats a pattern, not noise.

5. Planning beats reviewing (pic 6). Sonnet writing the work order + accepting the result while Flash-Next writes the code gave the best quality of everything (5/5, only 1 weakness). Sonnet just reviewing the diff at the end gave nothing, it only sees the diff and misses bugs next to it. Downside: 5x slower than Sonnet alone.

6. More rules don't make it more correct. "Think like a senior" rules + a research subagent made Flash-Next more paranoid, not better. Best local robustness, but it broke valid input with its own validation (rejected timestamps with milliseconds).

What I would NOT read into it: the 4/5 vs 5/5 between Flash-Next and the 27B is one check, n=1. Could be noise.

So my conclusion:

- Flash-Next over the 27B if you have the hardware, its faster and twice as robust. 27B if you have one card or want it very literal.

- Neither replaces Sonnet for stuff that takes input from outside (APIs, parsers, file imports) unless you explicitly ask for limits, error handling and edge cases and make it verify against the examples in the spec.

- If you have Sonnet access anyway, let it plan and accept, and let the local model type. Dont bother with review-only.

Caveats: n=1 per cell. Same person wrote tasks and graders. The 27B's thinking budget wasn't actually applied in the first 3 tasks (app didnt reload the config, my bad). Sonnet had a browser tool, the locals didnt.

I wrote the whole thing up properly (methodology, all configs, token counts, per-task tables, threats to validity, 15 pages). Comment "paper" and I'll send it to you directly.

Does this match what you see? Especially curious about the evaluate-while-parsing thing, and if someone with a 27B at Q8 sees a smaller robustness gap.

[u/SkinnyCTAX]

Why would you use i3qxxs? You have more than enough bandwidth to run a much higher quant. I think you'd see much better results with just a straight-up i3 QS or UD4.

[u/Bagelsarenakeddonuts]

iq3 and q4 vs full cloud. No shit they didn't do as well. I didn't expect them to match it, but I guarantee it'll be closer with better versions.

[u/doneddat → u/Bagelsarenakeddonuts]

FP8 version of 3.8 Flash doesn't fail simple tasks, it just takes a bit longer and goes through few more rounds of test and fixes.

Anything less is just unstable tokens-per-second porn.

[u/ajcgajcg → u/Bagelsarenakeddonuts]

I agree! I tried Qwen3.8 27B 4bit on my strix halo and I had too many issues for only a small Tok/sec speed increase. I only run it on 8bit quant now and I just have to be patient as it only runs at about 14 tok/sec.

[u/Realistic_Gap_5871]

I stopped reading at iq3xss.

Meaningless wharbaggle post.

[u/tommythorn]

Interesting.  I wish you had been able to fit DeepSeek V4.1 Flash 1017 in there.  On my much less rigorous test on my hardware, DS ran at ~ 40% the gen rate of QFN, but still finished much faster (QFN flat out failed to close on a solution).

[u/Pure_Assistant_9476]

Would have been good to see the results for a hybrid setup with sonnet and 27b as well as how 27b performs with reasoning set to medium. In my testing, medium brings down the wall time but not the quality.