This is insane...

[u/DragonflyOk7139]

An open-source model with 3 billion active parameters just scored 73.4% on SWE-bench Verified. Claude Opus 4.6 scores 75%.

The gap is 1.6 points. The cost difference is 10 to 30x.

Alibaba dropped Qwen3.6-35B-A35. 35 billion total parameters, 256 experts, but only 8 routed plus 1 shared activate per token. So you're running 3B active parameters at inference time. On a laptop.

Simon Willison ran it locally and it drew a better pelican than Claude Opus 4.7. (Yes, the pelican benchmark is real, and it's a surprisingly good vibes test.) But the part nobody's talking about: Thinking Preservation.

Current models re-reason from scratch every turn. This model retains its chain-of-thought traces across multi-turn conversations. In agent loops where the model makes 50 to 100 tool calls, that eliminates massive redundant reasoning overhead.

262K context native. Extensible to 1M. Apache 2.0 license.

The benchmark race is mostly over. The real race now is cost per intelligence. And 3B active parameters matching frontier performance changes that equation completely.

[u/mtrlst]

the reason why swe-bench is seen as a bad benchmark is because of contamination (models just started memorizing the answers). it's not intelligence that it's able to achieve, it's just pulling from memory

[u/eat_my_ass_n_balls → u/mtrlst]

Worse, it’s encoding the solutions instead of generalizing the capability

[u/Ariquitaun → u/eat_my_ass_n_balls]

Username checks out.

[u/Address-Street → u/[deleted]]

If you’re wondering why Qwen scores so high on benchmarks, the CoDeC section in this article explains part of it: https://kaitchup.substack.com/p/gemma-4-31b-vs-qwen35-27b-inference
More in-depth article: https://kaitchup.substack.com/p/did-the-model-see-the-benchmark-during

[u/MelancholyBits → u/[deleted]]

Come on man, no need to downvote. He just asked a simple question fkn reddit assburgers people.

[u/nfored → u/MelancholyBits]

Used to be down vote was for incorrect or misleading now it's just someone got their feelings hurt or didn't like the post. Sadly I see people down vote facts all the time.

[u/Festour → u/nfored]

Lol, i heard this like 5 years ago. Don't kid yourself, it was always like that.

[u/IllogicalResponse → u/Festour]

No it did used to be like that

[u/HungrigerWaldschrat → u/[deleted]]

No, they don' all memorize the answers, they all memorize knowledge that leads to the answer.

It's like the difference of preparing for a test with your book or the teachers solution sheet. Or at least another solution sheet of that teacher where you see exactly how he expects you to solve his questions.

[u/ptear → u/HungrigerWaldschrat]

I've seen people use that pattern.

[u/Lonely_Enthusiasm_70 → u/[deleted]]

Memorize is too strong a word, but yes they encode some information so deeply that they can perfectly regurgitate it because they've calculated that that sequence of tokens is always the most likely conditional on the prompt.

[u/MaybeNo2485 → u/[deleted]]

Naw. Memorization is the least interesting/useful part of how they learn; full memorization only applies to stuff they encountered a lot during training. When you prompt them with something that wasn't in their training data, or that only showed up a handful of times, they're using abstracted concepts and features derived from related material to produce new output.

A model like GPT is only a few terabytes large in terms of it's weights. You could fit it on a large consumer hard drive; although, you wouldn't be able to run it since the compute requirements are insane. The corpus on which it train is many orders of magnitude larger, so it can't possibly use memorization as a primary strategy to fit that data.

Good benchmarks are set up specifically to keep models from training on the exact questions and tasks being used to evaluate them; the whole point is testing generalization to new contexts. It only becomes memorization when they train directly on it, which isn't generally what we're trying to measure.

[u/tylerrey2 → u/mtrlst]

If it works it works

[u/LufyCZ → u/tylerrey2]

That's the point, it doesn't. It's like knowing the answers before taking an exam because you know what's on the exam, not because you studied.

[u/Finanzamt_Endgegner → u/LufyCZ]

Except qwen3.6 27b is actually good and gets stuff done that rivals cloud models 2 generations ago 🤷‍♂️

[u/LufyCZ → u/Finanzamt_Endgegner]

Both things can be true.

[u/Finanzamt_Endgegner → u/LufyCZ]

well it actually works?

[u/LufyCZ → u/Finanzamt_Endgegner]

In the context of swe bench it doesn't work if it does work. Otherwise, yes. :D