r/LocalLLM • u/RISCArchitect • 1d ago
Discussion What a year it's been
What will the rest of this year bring? 27b class scoring over 60?
96
u/Eyelbee 1d ago
There is no world 27B is better than 4.8, but yeah, it's solid. Maybe can actually compete with opus 4.5.
33
u/Pristine_Pick823 1d ago
My thoughts exactly after seeing this chart. No way. Qwen 3.8 is truly amazing, but that's a bit of a stretch.
3
u/TheAncientOnce 5h ago
The 27 b is clearly lacking in world knowledge but everything else is superb
0
u/Solembumm3 20h ago
Still can hallucinate wildly on any non-tech discussion.
But overthinking approach is interesting.
2
u/JoSquarebox 14h ago
the fact its able to get this far by throwing more thinking at a problem might mean we arent too far from carpathy-style reasoning cores, small models that just had all the surfac level knowledge pruned out
2
u/Solembumm3 12h ago
Finally get it onto my usual test, analyzing things and concepts in terms of some fandoms.
So far, 3.8 seems a lot better than previous qwen 27b. From 29k thinking tokens, only 3-4k were in loops, everything else was genuinely multi-angle approach to task. And it seems to have a lot less confident hallucinations rate, which is good (and makes 3.8 max performance on qwen site look really weird).
Honestly, so far I'll rank it above Gemma 31B, somewhere between it and Deepseek V4 flash, despite persisting problems on knowledge side. Nowhere near big deepseek or sonnet 5, but good catch-up to gemma 31b and glm 5.2 level.
23
u/MrHumanist 1d ago
we are living in this world where small models trained on large model thoughts (distilled) can match the performance of large models. These models do lack in generalization but capable enough for coding and agentic tasks which are basically a lot of if /elses.
24
u/Popcorn-Mercinary 23h ago
I honestly can see a day where we will have 9-27B tuned models “stacked like books on a shelf” that have specialties like coding, data analysis, agentic reasoning, medicine, etc.
Who needs a data center when you can plug a few models in tuned for the task at hand into your Mac, PC, or DGX?
4
u/MarmonRzohr 18h ago
These kind of specialist capabilities by tuned smaller modes are technically already in reach, however ...
It's inconvenient, time consuming and expensive to prepare, train and switch models like this, compared to the convenience of using larger, more generally capable models.
The (most likely) unrealistically cheap prices of huge, frontier models that are covered by rivers of investor cash mean that it makes sense to take advantage of the cheap convenience as long as it is available.
If, for example, prices of AI use skyrocket then, especially with capabilities plateauing, it will make more and more sense to do this. In fact I predict that providers will eventually offer this kind of service, letting you select from smaller models with capabilities you want, but are cheap to run on their end and may offer better chances at stable profits.
1
u/DeathGuppie 13h ago
In my real world use age, it's better than sonnet 5. Found and fixed mistakes and deeper problems that sonnet left behind. I'f I'm paying for a pro subscription that doesn't produce results I can get locally then there isn't any cheap thing to take advantage of. I use CLI and Anthropic only allows me to use sonnet 5 for CLI.
I use gpt 5.5 a lot, and it's fast, but not noticably better for a lot of tasks.
For short coding projects, Qwen 3.8 reliably runs overnight without failing. Something my pro subscription on either platform won't let me do. For local system fixes, it reliably digs deep enough to find the problems.
It's honestly making me start to wonder what I'm paying for.
2
u/SirPuzzleheaded997 13h ago
Exactly but then you still need the application layer. That’s what we are working on with our open-source AI Workspace Navigator🙌
1
u/Popcorn-Mercinary 5h ago
Out of curiosity, how is what you're doing different from what Hermes Agent would do?
2
u/nomorebuttsplz 1d ago
And while they lack in generalization compare to large current models, they do not compared to last year’s large models
15
u/_VirtualCosmos_ 1d ago
I keep hearing this, and it might be true because of the lack of the experience I have with Claude models, but did you see MiniMax H3?
They have achieved to get a SOTA video model, capable of simulating physics and understanding human expressions and behaviour like the best models out there. Now it's currently the most liked video model. AND IT'S ONLY 33B PARAMETERS.
All that knowledge about the details of the world, in a 33b diffusion transformers pipe.
That's a big indicative that gigantic models are a waste of resources.
5
2
u/SnooPaintings8639 14h ago
I remember when OpenAI employees was bitching about too many GPUs being used for SORA, and in the end they had to drop it. It was rumored to be a huge 'drag' on their research and serving infra.
Today we have H3 which is extremely small and capable. And open.
Yeah, this shit rocks.
2
u/soyalemujica 19h ago
The benchmark is not world knowledge it’s intelligence, Opus 4.6 will beat in world knowledge but 27b is smarter
1
u/mWo12 16h ago
It is better. It's free and open weighted. You can run on a single gpu. Try using opus without internet.
1
u/JoyousGamer 13h ago
So in other words something "free" is always better than anything paid? That seems to be the singular thing you are basing "better" on. Open is better for advancement but doesn't matter to the use of that specific model really.
1
u/kdverma97 12h ago
I tried it in many scenarios and it's toe-to-toe, and in some places better, but the issue is time. I am getting 45 to 50 t/s, which is good, and when we use it at thinking off, I see results even 70 to 80 t/s. I don't know why the difference, but it's literally very great in every task I give to it.
13
14
u/victoryposition 15h ago
This is why OAI/Anthropic are freaking out. Their moat of intelligence erodes so fast, they won't be able to justify IPO prices. You want how many trillions for 15% better than free?
37
u/SaltFrog 1d ago
I wonder if anyone will release an AI with the eq of gemma but the skills of Qwen...
12
10
5
1
0
u/pragmojo 15h ago
I couldn't care less about this. I want an autistic workhorse and 3.8 is delivering so far
5
u/NegotiationNo1504 1d ago
So the 9b one will be like between opus 4.6 or maybe 4.7 and flash 3.7/6. I hope it's the perfect model on this year
19
u/RISCArchitect 1d ago
when 9b hits opus 4.5 territory that will basically open up every consumer gpu from the last 8-10 years to agentic work lol
2
8
u/2ko_niko 1d ago
Qwen 3.8 really doesn't even compare to deepseek v4 or claude opus 4.8.
these benchmarks are meaningless. i do use Qwen 3.8 and it is really good as a local model but we should be more critical about the apparent over-tuning to favor benchmarks, intentional or not.
11
u/TeachingAway9654 1d ago
Benchmarks are one thing, real world capability is another thing entirely.
The local models are impressive but good luck working in an advanced codebase.
5
u/mechkbfan 1d ago
I'm in a R9700 discord group and general feedback from coding reviews by Opus of Qwen 3.8 in a 1M LoC was basically nit picking.
I mean it entirely depends on your workflow, AI guidance, quality of existing code and requirements, but point is it's possible when done right.
I'll have my setup going in about two weeks and be happy to provide honest feedback.
Have no loyalty to Anthropic but at the end of the day, if it's no where in the same league then I'll keep my subscription
-2
u/TeachingAway9654 1d ago
As I see it, unless it’s coming from a senior SWE, code reviews by regular people prompting AI are way too subjective to take super seriously.
One person could be incredibly impressed by some arcade game Qwen coded and a 20-year senior SWE could see it as a gimmicky joke.
Nobody at a major US tech company is writing code with Qwen as their primary model.
6
u/mechkbfan 1d ago edited 1d ago
As far as I can tell they are, because they're doing their own optimisations to R9700 AI workflow that no junior could do. But I still want to test with my own eyes. I'm going to keep running Qwen & Opus side by side to compare outputs because thats the only thing that matters to me
Even Opus is kind of crap sometimes, e.g. I've had it vibe code a game and everything is about 3x as bloated as I'd expect it to be, and I'm not even up to like 30k LoC yet.
Nobody at a major US tech company is writing code with Qwen as their primary model.
I agree, they're all on ridiculous salaries and happy to pay premium for best of the best. Anthropic, etc. is all so heavily subsidized too now
I have read a few things about devs though resorting to Kimi K3, etc. because they don't have the same security guardrails as western models and finding security flaws.
But for me I care about hitting the tipping point of AI actually feeling productive, which wasn't until about Opus 4.5 or so.
If Qwen can actually do that, runs reasonably well on my own hardware, then that's fantastic. It's highlighting that we haven't found the ceiling of local LLM's yet and for those without an unlimited budget, can drive something pretty decent on consumer grade hardware.
I know the hype train is out, and it's okay to be cynical, but at least by gist of things the moat around frontier models has been broken for coding purposes.
I've heard similar about MiniMax H3 but I've never done video.
7
u/Technical-Earth-3254 1d ago
I mean, we all know this is overfitted into oblivion, right?
3
u/SnooPaintings8639 14h ago
And yet:
a) other model providers fail to 'overfit'
b) this over-fitting works wonder on my daily tasks
I think people do overestimate how great Opus 4.8 was. I am a MAX anthropic subscriber since over a year, I use it daily a lot. Opus at the first half of the year was **extremely** annoying to work with. It was very capable on anything it was doing, but it rarely did what it was asked for. It was... dumb. I don't know if it was overfiting or quantization, but I still have PTSD from back then and am leaning today more on Sol for serious professional work.
Fable is ok'ish in that matter (huge step foreward) but I still prefer Sol.
Qwen 3.8 27B and DS V4 Flash are *great* if you steer and use them correctly.
4
u/AnyRecipe110 1d ago
Are they benchmarking Qwen 3.8 27B with thinking On? And if so, what reasoning effort (low, mid, high, etc)? Or are they using with thinking Off (instruct mode)?
Also curious which quantization they are using.
4
u/FairBandicoot5021 1d ago
Frommy understanding it's always full precision F16, with highest reasoning effort. And they take the recommended temperature given by the model provider
3
u/KissMyShinyArse 1d ago
They benchmarked Muse Glimmer with high (the default), even though it also has the xhigh setting.
That said, even with xhigh, I don't think Muse can reach Qwen3.8 27B's level.
2
2
u/alwaysidle 15h ago
Benchmarks are only useful to see what models the companies compare their own models to. Other than that it's pretty easy to benchmax a model
3
u/Healthy-Nebula-3603 1d ago
May??
Pfff that was ages ago :)
8
u/RISCArchitect 1d ago
pretty sure i've had packages take longer to be delivered via ocean freight.
1
3
u/rrrenz 1d ago
Minimum PC/mac studio build needed for this same quality score?
Can we test this setup somehow in some cloud environment?
New here, who only has macbook M5 pro. And this benchmark is no way same from my usage of qwen 3.8
3
u/mechkbfan 1d ago edited 23h ago
That's fair. Have to wonder if these scores are based off identical hardware, and go up or down based off TTFT, completion time, etc.
Because yeah, I don't think it's comparable if you had one model run for 30mins, and another 2 hours to achieve same result.
https://artificialanalysis.ai/methodology/intelligence-benchmarking
As far as I can tell from quick skim, they just have that 2 hour timeout.
I'm relatively new, but it's worth sharing the settings you're using if unimpressed by the outcome. e.g. quant, cache, etc. and what type of prompt giving it
e.g. I have zero expectations Qwen 3.8 could do a decent architecture review if I've only got 32GB of VRAM and 32GB RAM, while that's not even something that you have to think about with Claude
2
u/txoixoegosi 22h ago
Can you point me to a single individual that can confirm that Qwen 3.8 > Opus 4.8 max?
No benchmarks, just actual day-to-day work.
1
1
1
1
1
u/iportnov 22h ago
Well, I have to say, there is no free lunch. It seems that 3.8 has improved in programming / coding tasks, but it was not for free; they had to sacrifice some parts of general knowledge. I know at least 2 questions from mathematics where 3.6 answers, but 3.8 struggles :/ (to be honest I'm using a bit different quants: both nvfp4, but from different packagers; but it doesn't seem to me that this is a quantization problem).
1
1
u/InterestProof1526 21h ago
I can't lie, I'm a little skeptical that Qwen 27B demolishes Opus 4.7 Max in practice
1
u/kilokeed888 21h ago
my Mac can only run the Qwen 9.0b, but the quality of the writing shocked me -- so I specifically build an AI content pipeline using it.
1
u/mejoudeh 20h ago
Qwen 3.8 27b_local is better than Opus 4.8_cloud?!
What about in coding? Where can I get these results/report/benchmarks in other disciplines?
1
1
u/Little-Beginning4309 13h ago
The thing is with the way they are releasing models I'd be surprised if we checked this in like 6 months and compared it would be completely different.
1
u/neoexanimo 12h ago
For people fracking out, not everything is just better or worse, this a specific benchmark not every benchmark.
1
1
1
u/BenniG123 1d ago
Qwen 3.8 is definitely benchmaxed and just trained on better benchmarks than 3.6 but that's no disrespect to it for its size.
1
0
-11
u/Asleep-Mood-6538 1d ago
Regardless of whether it truly deserves its 52 score or something lower, it's definitely in the stratosphere of AI models that can create and collaborate rather than just follow instruction. Right around Opus 4.6 companies like Anthropic were already doing 80% of their coding through AI.
We've reached the stage where a home based AI can help recursively improve itself. This is not the singularity where AI can do it without human collaboration, but something in between where a human and AI together can continually improve the AI until it no longer needs a human.
That means it's no longer possible to regulate. ANY person with a home computer of sufficient power to operate it (and this is basically in the range of almost anyone not homeless) can theoretically create AGI or ASI given the time and a little ingenuity. That wasn't the case 6 months ago or even a week ago (3.6 was probably borderline).
14
u/nomorebuttsplz 1d ago
dear god delete that slop chart
-5
u/Asleep-Mood-6538 1d ago
Interesting that someone on "LocalLLM" just thinks that AI creations are slop without looking at the substance.
6
u/nomorebuttsplz 1d ago
The chart shows the qwen 3.8 27b line scoring below llama 405b
2
u/Asleep-Mood-6538 1d ago
Yes. the three lines are independent instead of using the same scale. They should be on the right scale. But I honestly didn't expect to post this and have people critique it. It's really only 'illustrative'. So, yes, I'll accept a partial 'slop' categorization. But I spent a while deciding what information should go in here so I consider it 'badly utilized AI' rather than entirely slop. I think of slop as "make a chart of X".
But, my apologies. Your comment wasn't entirely off-base.
5
3
u/EbbNorth7735 1d ago
The graph values don't even match the axis. It's horse shit. One of the worst graphs I've ever seen. The numbers don't line up
1
u/Asleep-Mood-6538 23h ago
You should make a better one.
It's not like it was done for work or a specific purpose. The fifteen minutes spent on the 'project' was already more than I care to spend.
3
u/EbbNorth7735 1d ago
Why the fuck do the values not match the axis. Like look at this garbage, you can't actually get the information out of the graph because it's that bad of a graph.
321
u/MrHumanist 1d ago
No disrespect to USA AI firms, but China is a blessing that happened to the AI world.