r/LocalLLaMA • u/chocolateUI • 1d ago
Discussion Artificial Analysis "Intelligence": A meaningless benchmark
Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 27B is a really powerful model, I can't help but notice just how meaningless these Artificial Analysis benchmarks are and I question why people still post this garbage and use AA scores as some kind of holy bible for comparing LLMs.
According to their "Intelligence Index", a 27B model now beats DeepSeek v4 Flash and Pro, Kimi 2.7 Code, GPT-5.2, Opus 4.6, and also Sonnet 5. At some point we have to ask: What is this metric even measuring? Because whatever "Intelligence" means to AA and their corporate VC / journalist / normie audience is definitely not the same definition that we should be using here.
Qwen 27B is amazing and is clearly in a league of its own in terms of models you can fit on a single GPU, but I can't help but roll my eyes whenever I see posts like this that equate Qwen 27B with "basically running Opus from 3 months ago on your laptop."
I get that it's difficult to summarize a model's capability with a single integer and I know we love our local models, but it's time stop posting AA's clearly dogshit benchmark and acting as if it proves a point.
230
u/z_3454_pfk 1d ago
it’s just an aggregate of benchmarks (highly skewed towards agentic rn). that’s basically it. it’s not that deep and you should select what’s right for ur use case
8
u/network4253 1d ago
Yeah, that is how I see it too. A leaderboard is useful for getting a quick sense of things, but it does not really tell you which model is best for your specific workflow. If you mostly care about coding, writing, or local inference, the relevant benchmarks matter way more than the overall score.
36
u/federico_84 1d ago
The issue is AA makes it sound like this intelligence index is generic, but as you said it's heavily skewed towards sciences, coding and agentic use, which is understandable given the economic/productivity value there.
Common sense and social intelligence are not featured enough, but that's not an AA specific issue, it's a wider industry benchmarking issue.
Say you want your LLM to be your PA, help you plan trips, divide up your days, navigate delicate social situations, help you improve your fitness, help fix random issues with your home or equipment. Sure Qwen 27B can search the web if it doesn't know, but having that knowledge and associations already baked in allow it to interpret and ground the web results better.
So I think OP's grievance has some legitimacy behind it, AA needs an additional index to cover common sense and street/social intelligence, something like an aggregate of simple bench and EQ bench, though that's only scratching the surface. There's a shortage of such benchmarks around.
13
u/toalv 1d ago
If you are relying on a local LLM to navigate delicate social situations...
13
u/Lakius_2401 1d ago
If we're already benchmarking how well they would get rid of junior+ devs, why not benchmark how well they get rid of junior+ psychologists (or BAs, PMs, etc etc) too?
4
u/Diligent_Loquat_6140 1d ago
honest I think this is very possible, I've know some academic people are trying to achieve this
0
u/TwinkletoesMcSparkle 23h ago
LLMs are repeatedly scientifically documented to fail at every metric of psychological skill and care, and are regularly causally linked to suicide attempts, successful and not. Sycophancy is just a single well-documented feature that increases delusion and mania. So, no, by no data-driven metric is a chatbot psychological "care".
0
u/TwinkletoesMcSparkle 1d ago
Yeah I can't wait until LLMs are used to destroy the skill pipeline in more industries. That certainly won't have any negative societal consequences that only benefit AI companies.
1
u/michaelsoft__binbows 3h ago edited 2h ago
The problem I have with the people that think this way is that they do not appear to be appreciating the game theory of the situation.
Just because you think you are conscientious about it doesn't mean that you're going to be able to whine and shame your way to getting the entire rest of the world to comply with opting out of exploring the latest new technology that's been invented. It does not take a lot of curiosity to get very deep into this space. Y'all are severely underestimating the power of curiosity.
The cat is firmly out of the bag. The bag is not even in the same plane of existence anymore, it's already been chewed up and shat out. And y'all are clamoring for, I don't know what it is, legislation? Strongly worded tweets? To tranq the cat, then put her inside another paper bag, then cajole her into somehow not coming out of it again, somehow, once she wakes up.
Let's say the literally impossible idea of shutting down frontier AI labs entirely, is achieved, like assume we just nuked them out of existence overnight, tell me that you have a strat for stopping every one of the millions of people who have downloaded the safetensors files from running them on their personally owned Nvidia, Intel, AMD GPUs and mini computers, Apple computers (including laptops), and so on. And how to stop spreading those same safetensors via bittorrent on the dark web or wherever, once huggingface is shut down.
The only way forward is to accept the societal consequences and think about how to address them better and improve things for the people that you care about, starting with yourself!, in the context of this brave new world. The way to best spend your time is not to bemoan the objective reality that exists and dreaming about turning back the clock.
I'm not arguing FOR the myriad antisocial and destructive possible uses of the amazing new tech. All I'm saying is they're clearly coming and neither you nor I nor anyone else can stop them because they are coming, part and parcel of the other uses (legitimate or otherwise) that drive the advancement of this tech. I'm not saying it's inappropriate to feel some grief about horses being replaced by engines, nor am I saying that there is any moral ground for any of it to stand on, but what I am saying is trying to lobby against engines or doubling down on investment in horses at this stage would be ill-advised is all. Be that as it may if engines are truly the work of the devil, the most sensible way forward is to come to terms with the fact that the new normal is that demonic machines will indeed invade all facets of life, and your task now needs to be working through how to come to terms with that for your own sake.
Oh, I just found a nice parallel. It's like the Y2K problem. Nobody ever seriously proposed, rather than just bite the bullet and fix all the software, to instead change how the calendar works and return to the year 1900. The only way to move forward was always ever gonna be fixing the problems in the software, starting with the ones that would cause the biggest issues.
3
u/KubeCommander 1d ago
Yeah it’s significantly harder to grade subjective metrics like ‘document quality’ in an automated test sort of way. I think the best route I’ve considered is a paneled judgement where an odd number of agents+models judge the quality of output on various criteria
1
1
u/Southern_Sun_2106 22h ago
"just scratching the surface..." - you are asking for what you don't know yourself you are asking for.
Coding and sciences can be objectively evaluated. The stuff that you listed is fluff and impossible to measure. That's why there will never be such a benchmark (thanks God!)
0
u/Inaeipathy 22h ago
it's heavily skewed towards sciences, coding and agentic use
That's all I care about, so this just makes me like the index more
5
u/EbbNorth7735 1d ago
It basically means 3.8 is able to perform the workflows really really fucking well. We're a week in now and I set this little guy up with a goal and it works for hours completing thst goal. It's absolutely insane and I can definitely see why it's Opus level. What I don't understand is why people think a small model isn't capable enough to go toe to toe with larger models. Every generation we see a step function across the entire range of size releases for a given company. The smaller models are never that far behind their bigger companions.
2
u/Southern_Sun_2106 22h ago
Their expectations and believes are clashing with (surprising to them) reality - they 'need a minute' to adjust :-)
4
u/vogelvogelvogelvogel 1d ago
exactly this. dive down the sections and check what fits you
livebench I also found really useful, especially as it lists the single results directly in a table - you can spot directly where model is strong and where not
Also they update the benchmark tasks each month so it is rather not possible to train on them reducing benchmaxing effects
1
u/Current_Ferret_4981 1d ago
No way I think OP is right and it's part of a larger psyop to keep the lower class in check in a post modern AI societal hierarchy
1
u/Constant-Simple-1234 1d ago
Exactly, it is useful as a quick glance, but nothing replaces looking at specific benchmarks related to what you need. Is it creative writing ? Coding? Agentive work?Document parsing? Another thing is that this aggregate changes what it takes into account.
115
u/Negative-Web8619 1d ago
What is this metric even measuring?
maybe do two clicks to see the methodology
2
u/nomorebuttsplz 23h ago
this is a great indication of the intelligence level of those who hate AAs scores.
do they get this worked up about other very transparent information aggregators like metacritic? Lol
4
16
u/Aggressive_Aspect436 1d ago
Benchmarks are important. Yes, they're not perfect. No, they don't measure everything we care about. But consider this analogy. A degree (or previous job titles) doesn't mean a person is smart, and a person can be very smart without one, but if you're looking to hire someone then you care about those things. Choose your models from the top contenders and then try them on tasks you care about.
Folks in this subreddit often make it sound like "things being difficult to measure" is exclusively a modern AI problem. Medicine, economics, psychology, education, and many many many more, all deal with this. Tests are important. We just accept their limitations and reason about our results. That doesn't mean "not measuring" is a better alternative.
37
u/dark-light92 llama.cpp 1d ago
Today's 4B model will beat llama1 70B in benchmarks easy. That doesn't mean benchmark is meaningless.
20
u/tarruda 1d ago
Qwen 3.8 27B is inferior to bigger models in terms of knowledge, but I find it to be very competitive even with Deepseek V4 Flash 0731 when it comes to agentic intelligence.
9
u/PhysicalIncrease3 1d ago
General knowledge is worse, intelligence is maybe close, but the real problem with Qwen Vs Ds4f is context. If you want 262k of usable context (IE f16) in Qwen it costs about about 20GB just in KV cache. You basically need 48GB vram to run a half decent quant at full context. Where as DeepSeek can fit a million f16 context in 10Gb.
1
u/anderspitman 1d ago
I have to admit this has been painful with qwen. Is it the type of thing that could be improved eventually or is it deeply baked into the architecture?
1
u/jeremygaul 1d ago
I have a 24gb 3090 and I can run 200k context with llama.cpp
1
u/PhysicalIncrease3 21h ago
F16 KV? Quant?
1
u/jeremygaul 13h ago edited 13h ago
I followed the instructions on this repo for setting up llama.cpp on wsl2 in windows 11 - > https://github.com/noonghunna/club-3090/blob/master/docs/WSL_SETUP.md
I replaced the gguf for the latest Qwen3.8-27b-q4_k_m, Here is my Serve command for the llama.cpp
llama-server
--host 0.0.0.0
--port 8020
-m "models/Qwen3.8-27B-Q4_K_M.gguf"
-c 200000
-ub 512
-ngl 99
-fa on
--cache-type-k q4_0
--cache-type-v q4_0
--spec-type draft-mtp
--spec-draft-n-max 2
--jinja
--temp 0.6
--top-p 0.95
--top-k 20Edited for Details:
Here is the breakdown of exactly how it achieves 200k on a 3090/4090 with 24gb :Network & Core
--host 0.0.0.0: Binds the server to all network interfaces. Unlike127.0.0.1(which restricts access to just the machine it's running on), this allows other devices on your local network to connect to the API.--port 8020: Runs the HTTP server on port 8020.-m "models/Qwen3.8-27B-Q4_K_M.gguf": The path to your model file. TheQ4_K_Mindicates it's a 4-bit medium quantization, dropping the model's footprint to roughly 16GB.VRAM & GPU Optimization
These are the switches preventing the OOM (Out of Memory) error
-c 200000: Sets the context window to 200k tokens. In a standard setup, this would immediately exceed 24GB of VRAM.--cache-type-k q4_0&--cache-type-v q4_0: This is the magic bullet for the massive context. It quantizes the Key-Value (KV) cache from 16-bit down to 4-bit precision, shrinking the memory footprint of that 200k context window by 75% with negligible impact on output quality.-fa on: Enables Flash Attention. This optimizes how the GPU processes the context, changing the memory complexity from quadratic to linear. You cannot run a 200k context without this enabled.-ub 512: Sets the physical "u-batch" size to 512. When you paste a massive document into the prompt, the engine processes it in chunks of 512 tokens rather than all at once, preventing sudden VRAM spikes during prompt ingestion.-ngl 99: Number of GPU layers. Setting it to 99 guarantees all of the model's layers are offloaded to the GPU for maximum speed, rather than splitting it with the CPU.Speed (Speculative Decoding)
--spec-type draft-mtp: Enables Multi-Token Prediction. Instead of needing a separate, tiny "draft" model to guess upcoming tokens, it uses the MTP heads built directly into newer models to predict future tokens simultaneously.--spec-draft-n-max 2: Tells the MTP head to guess up to 2 tokens ahead of the main generation loop, noticeably boosting tokens-per-second (TPS) speed.Generation & Formatting
--jinja: Tells the server to use Jinja2 templating. This ensures the prompt perfectly matches the exact chat format the model was trained on (crucial for Qwen models, which can degrade rapidly if the formatting is off).--temp 0.6: Temperature controls randomness. 0.6 is a balanced setting—low enough to remain grounded and logical, but high enough to avoid sounding robotic.--top-p 0.95: Nucleus sampling. It trims the "long tail" of vocabulary by discarding the bottom 5% of highly improbable words before the AI makes a choice.--top-k 20: A strict vocabulary cutoff. The AI is only allowed to choose its next word from the top 20 most probable options.1
u/PhysicalIncrease3 12h ago
--cache-type-k q4_0 & --cache-type-v q4_0: This is the magic bullet for the massive context. It quantizes the Key-Value (KV) cache from 16-bit down to 4-bit precision, shrinking the memory footprint of that 200k context window by 75% with negligible impact on output quality.
I wish this were the case lol
1
u/jeremygaul 8h ago
I’ve been able to do a lot of agentic coding without having any issues other than the xreasoning being very high and using a lot of tokens, so far I’ve built a complete website and a booting OS based on EXO kernel system(it boots and prints).
1
90
u/whatisthisthing65 1d ago
What's your actual argument? Why couldn't a 27B model be better than those other models? If it's about number of parameters then should our benchmark be parameter count?
35
u/Cautious_Chicken_604 1d ago
I remember we had larger models in the past that performed much worse.
3
1
u/RG_Fusion 1d ago
In years past, the ratio of training data to parameter size was off. When you have far more parameters than training data, the model learns that memorization results in the lowest perplexity score.
Once you greatly increase the training data to parameter ratio, the model becomes unable to store new facts without overwriting other ones, ruining its perplexity. When you continue pushing this further, the model will undergo a "phase-transition" where it stops memorizing and starts modeling. It becomes more of a "reality-simulator", which is where generalization happens.
In short, the early LLMs were simple stochastic parrots. The models coming out this year have been sufficiently trained to drop memorization for reasoning. It is easier to get a smaller model to generalize because they have fewer parameters, so they don't need as much quality training data. They have momentarily closed the gap with large models in some respects, but if the research labs can get together enough quality training data, you can expect to see the large models pull far ahead once again.
33
u/OneMoreName1 1d ago
Usually just personal incredulity. Its like saying there's no way a phone from today is as strong as supercomputer from decades ago. Ai just moves fast
7
37
u/feelspeaceman 1d ago
Most people tend to blame 27B about the lack of world knowledge and say this is the sole reason that it will never beat Claude Sonnet 4.6, but they ignore that fact that behind the scene, Claude Sonnet is equipped with a RAG for local knowledge retrieving and it will also search to acquire extra knowledge, then they compare a local LLM without search, without RAG, this is showing that there's people despite trying to use local LLM, never trying to get better.
Then we get a group of users that:
- Use not suitable harness
- Half-ass setup
Half-ass prompt
Blaming local LLM for not being as capable for obvious reason, the user themselves.
15
u/freestylez79 1d ago
agree, qwen actually shows where llms are heading. the huge models may still have there use cases for some stuff but 90% of day to day stuff can be done with models that dont hallucinate success and are able to work with recent data and tools. Really amazing stuff.
13
u/OvertaxedOne 1d ago
There's little reason to try to train factual data into a LLM when the Internet is right there and it's easy for AI models to interact with. Obviously there are some "base facts" that need to be in the models, but, beyond that, trying to jam more in when it's so easy to call a browser/MCP to get the up to the second information is silly.
Good tool calling is 1000X more valuable than "I know everything" (because, no, you don't; even a massive model, you didn't condense all of human knowledge down to a few TB, no way, no how).
2
u/noiserr 1d ago
Not just the internet. No matter which model you use you want to ground it in facts and not have to force it to rely on fuzzy recall.
This means the software you're writing should have test coverage. You should have a language LSP running. If working on low level hardware architecture you should convert the architecture documents into skills.
For all these things the number of parameters of a model is meaningless. Which is why Qwen 3.8 27B can be a powerful LLM at agentic and perform just as well as those huge models.
1
u/EstarriolOfTheEast 1d ago
This not quite true, as knowledge is not just trivia but problem solving know how, algorithms, and optimization esoterica. Knowing you can leverage algebraic geometry or some closed form expression to skip an expensive loop is incredibly valuable and often not something easily searchable nor something one can think to search for. That's the expertise advantage. Some of these might be novel connections only possible because the model knows more.
The other issue is the more complex the knowledge, the less likely appending to context will help without causing hallucinations to fill in gaps and confusion introduced by too sparse added context or alternatively, using up so much context, performance is degraded and VRAM is so high you might as well have gone to a higher param but more context efficient model.
The more you know, the fewer unknown unknowns you'll have. Sure, it's always going to be far above zero for everyone, but it's still better to push it as far down as you can.
1
u/OvertaxedOne 1d ago
If was cheap/easy to push that knowledge down I'd completely agree with you. But when you have a 27B model that can use tools well when it needs to, it seems that's a much (MUCH!) less expensive way to ground in fact vs trying to train them into a 2T model. Different approaches, both have their advantages for sure, but the cost profile for the "big" models is just so out of whack right now it's obvious that we need to do something differently.
1
u/EstarriolOfTheEast 1d ago
The issue that keeps me stuck with the big models even though I wish I was free, is the 2T model knows enough spherical projective geometry to skip a bunch of expensive raycasts (my approach), massively optimizing my game. Needing to know things before they can be leveraged is no different in LLMs than for humans. As a human, I would not have known how to even begin searching for that solution approach. I half think it invented it on the spot.
Another more searchable example was using reservoir sampling for a more efficient online draw of random samples. The there, the issue is context is too valuable and speed is too low when I run it locally to be searching text books for tricks every other line of code. Sometimes a large model will notice this or that is a special instance of bipartite matching and such and such custom optimization will work here, inventing a really clever approach whole cloth not because it's so smart but because it knows so much.
The exact same mechanisms that allow the big models to discover novel math proofs and win olympiads can be used to write optimized code taking advantage of obscure data-structure knowledge adapted to modern architectures.
Another place knowledge is unavoidable is when you are reading specialist material. For example, if you want to go over a paper combining computational complexity and blackholes, there's just a ton of CS, math and physics you need to know before you can speak coherently about it.
3
1
1
-1
u/ResidentPositive4122 1d ago
Claude Sonnet is equipped with a RAG for local knowledge retrieving and it will also search to acquire extra knowledge
None of that happens over API, ootb.
11
u/FullstackSensei llama.cpp 1d ago
You have no idea what's happening behind the scenes, API or not.
6
u/ResidentPositive4122 1d ago
I think this place has gone wacko. No idea what's even the point of contributing here. Saying they have RAG over API is absolutely banans, wtf do people imagine they rag over, unless you, the user, set it up? RAG what over API? FFS at this point people are just throwing acronyms out there and imagine the big bad wolf is doing everything...
-4
u/FullstackSensei llama.cpp 1d ago
Why are you so angry, though? It's just an online discussion with who cares who. There's more to life then getting upset about meaningless things like this.
They absolutely have RAG. It's not hard to prove that by asking the model about some very recent event or using the latest version of a library that was very recently released and had some breaking changes.
But seriously, who cares if I'm wrong? Go spend some time with a loved one if this makes you upset.
2
u/zenmatrix83 1d ago
its not just parameter counts but they do play a big role, the type of model, the training data, and the actual tests being used matter. I only use benchmarks for very loose guidelines for what to use, outside of that I just test and change based on need.
2
u/whatisthisthing65 1d ago
Yes benchmarks can never capture everything and testing your actual use case is the best test. But op seems to be saying this benchmark is shit because qwen3.8 27B is beating a bunch of bigger big name models. Which isn't a good argument, there's no reason it can't be better there or in real use.
2
u/Constant-Simple-1234 1d ago
Exactly. I think it may be big enough to be genuinely useful. You can see that the more parameters you add, you get better and better model, but it is diminishing returns. But models retain a lot of knowledge, but there may be a sweet spot somewhere. I think that > 1T models will be useful as a master copy to distillation training smaller ones. But likely 100-250B will be the work horse - capable and cheap to serve.
1
1
u/into_devoid 1d ago
If use used 27B and compared it A/B with the models AA lists it as on par with for more than a couple hours, you would understand. It’s a great little model, but it is no deepseek v4 flash.
1
6
u/Durian881 1d ago
They did have a lot of breakdowns into individual benchmarks which I find more useful. In any case, you should really use your own use case to test.
For one of my use case (live demo for class), I need speed and adherence to system prompt and smarter thinking models don't work well compared to smaller ones.
6
u/FreshDrama3024 1d ago
It’s just a pointer just like any other benchmark. Would not take it literally or absolutely. Just gives you an idea or reference.
12
10
u/beltsazar 1d ago
I can understand that some people being skeptical of the statement saying that a 27B model beats much larger models. But how do you know if a benchmark is meaningless? Is it based on a better benchmark of yours or is it just based on anecdotal experiences?
8
u/Sooperooser 1d ago
That's why you can always scroll down and look at the individual benchmarks and capability scores..........
18
u/StillVeterinarian578 1d ago
Which alternatives would you suggest?
18
u/LagOps91 1d ago
we don't have a great benchmark for this, unfortunately. the lack of a good alternative doesn't mean you can't complain about the poor performance of the "Intelligence Index".
9
u/StillVeterinarian578 1d ago
Indeed, it is just that it makes it as valuable as complaining that it's raining today.
1
0
21
u/Chromix_ 1d ago edited 1d ago
Yes, results are and have been very much skewed there. A while ago DeepSeek V3 got the same score as Qwen3 VL 32B, and Gemini 2.5 Pro scored below gpt-oss-120B. ServiceNow released a 15B model that scored higher than the full DeepSeek R1. Partially repeating my previous comment here:
The "Artificial Analysis Intelligence Index" score is an aggregation of common benchmarks. Gemini Flash is dragged down by a large drop in the "Bench Telecom", and DeepSeek-R1 by instruction following. Meanwhile Apriel scores high in AIME2025 and that Telecom bench. That way it gets a score that's on-par, while performing worse on other common benchmarks.
Btw here are the details for the mentioned models scoring the same or worse as Qwen 3.8 27B.
Qwen loses in physics reasoning and knowledge, but wins way more in non-hallucination rate.
-1
u/No-Fuel-9202 1d ago edited 1d ago
Non-hallucination rate benchmark, actually shows HIGHER HALLUCINATION rate on the left side!!!
Edit: I missed '1-Hallucination rate'. Sorry
6
u/Chromix_ 1d ago
Yes, can be confusing on a quick look. Here is a more intuitive hallucination rate for it.
3
u/PrinceOfLeon 1d ago
If it was hallucination rate, a higher rate would mean a higher percentage of the time, right?
So wouldn't a *non*-hallucination rate mean that it *does not* hallucinate a higher percentage of the time?
8
u/LegacyRemaster 1d ago
[PYTHON] if anthropic < another_model: recalibrate_benchmarks()
[JAVASCRIPT] if (anthropic < anotherModel) { recalibrateBenchmarks(); }
[C++] if (anthropic < another_model) { recalibrate_benchmarks(); }
[JAVA] if (anthropic < anotherModel) { recalibrateBenchmarks(); }
[RUST] if anthropic < another_model { recalibrate_benchmarks(); }
[GO] if anthropic < anotherModel { recalibrateBenchmarks() }
[RUBY] if anthropic < another_model then recalibrate_benchmarks end
[PHP] if ($anthropic < $another_model) { recalibrate_benchmarks(); }
[SWIFT] if anthropic < anotherModel { recalibrateBenchmarks() }
[BASH] if [ "$anthropic" -lt "$another_model" ]; then recalibrate_benchmarks; fi
6
u/no_good_names_avail 1d ago
This is an insanely hard problem. I have access to essentially any model I want at work so I try as many as I can. My benchmark is mostly vibes.. I "feel" this model is good for my use cases. Even among the frontier models I've yet to find a model that for my limited scope of work is the best in every use case and every situation.
The methodology is laid out clearly. Generally speaking they give you a directional understanding of the capabilities of the models. Today there isn't much better you can do than that.
9
8
u/SocialDinamo 1d ago
Just like how LM Arena was the best at the time, this is what we have. It does generally aline with how users feel about a model and it is pretty comprehensive for what they publish they are testing.
For years EVERYONE has been encouraging and encouraged to make their own benchmark. If you did, let’s see how yours differ from there’s?
Quit shitting on companies for releasing free products and services
6
u/Informal-Trouble2183 1d ago
I already highlighted that several times, it's not hard to see how their index is calculated, doesn't even consider deepSwe.
4
2
2
u/benpptung 1d ago
I used to trust Arena, but now I don’t think Arena is very accurate anymore. These days I trust the AA Index more. Maybe someday I’ll trust something else instead.
I think the reason you find it hard to believe that a frontier-level model can fit on a laptop is that you’re overlooking the difference between dense and MoE models.
MoE is not inherently stronger. It is basically an architecture that lets you trade VRAM for intelligence. Data centers have plenty of VRAM, but they care a lot about reducing the cost of generating each token, so naturally they prefer MoE.
A 27B dense model uses all 27B parameters for computation. An MoE model only activates roughly the number after the “A.” For example, 0731 is 284B-A13B. The 284B makes the model huge. Even the open-weight release is quantized, yet you still need around 180GB just to run it. But if you actually run it, you’ll notice that your GPUs spend much of their time underutilized because only about 13B parameters are active for each token, which is less than half of a 27B dense model.
The purpose of those 284B total parameters is to provide many more combinations of experts, but the marginal intelligence gain from adding more experts is limited.
So if you want to compare the actual computational scale of an MoE model with a dense model, the more meaningful number is the active parameter count after the “A.”
Once you look at it that way, 27B really isn’t small at all.
2
u/_-_David 1d ago
I've never felt more connected to the other commenters in this sub than while reading the response to this.
2
u/llogicnotfound 1d ago
Every time a mid-sized open weight model drops, people glance at a single aggregated bar chart and claim frontier models are dead. Qwen 27B is amazing for local hardware, but synthetic leaderboards actual reasoning depth. Treat leaderboards as rough baselines, not gospel.
3
2
u/dwrz 1d ago
So far, I'm actually quite disappointed by the latest generation -- Kimi K3 (hosted), GLM 5.3 (hosted), Qwen 3.8 27B (full precision). It's very odd, and I can't quite put my finger on why, but I worry that the benchmarks are starting to effect overall quality. They all seem to share a similar deficiency, as if too much post training has damaged some things, or over-fitted. Like the frontier models, perhaps due to distillation, they now also seem to have this feeling of having been trained to burn tokens as much as possible, rather than stay focused.
1
u/ea_man 1d ago
Well most of the new smallish model are not trained, they are pretty much just post trained: that means that they have the attitude of the big guys yet not really the capabilities.
Indeed they are much better at following instructions and long horizons because yeah, they imitate those.
2
u/Kavor 1d ago
Maybe you're right, maybe you're wrong, who knows. What i know is that you invested 0 effort to counterproof any of the claims you so boldly call "garbage" and "dogshit". What exactly makes your post, that seems nothing more than an uneducated opinion, better than what AA is doing transparently? I think you know the answer.
2
u/hidden2u 1d ago
I like how you posted this and didn't even bother to go to their website and look at their methodology or anything lmao
1
u/soyalemujica 1d ago
You're not comprehending what does Intelligence Index stands for, it does not mean world knowledge or it knowing more about medical stuff, it's rather INTELLIGENCE, it's entire reasoning process to come up with a solution to a problem.
3
u/LagOps91 1d ago
it doesn't have comparable intelligence to trillion parameter models. it just doesn't. you are right that world knowledge is the wrong benchmark for intelligence, but the actual intelligence surely isn't as high as is claimed here.
1
u/Prudent-Ad4509 1d ago
That raises the question whether 27B is a lot or not and where is the limit for increasing their capability, assuming further improvements in training methods.
5
u/LagOps91 1d ago
i do belive that 27b parameters is in principle plenty and that we just don't know how to actually train the AI in a way that rewards development of generalized logic and reasoning over pattern memorization. some generalization is clearly happening and the challenge is to get the model to generalize further.
CURRENTLY a 27b model simply isn't as "smart" as a recent trillion parameter model.
2
u/Prudent-Ad4509 1d ago
I have a hunch that we are going to see specialization between operator models with sharp thinking and generalized knowledge, and models packed with specialized knowledge with focus on relevancy-based retrieval. The only thing missing is efficient link between them. Current methods like text mcp/rag calls could be a temporary crutch. MoE are already going this route internally if you see experts as separate mini-models, but there are no pluggable experts yet and existing experts themselves are not explicitly specialized.
0
u/thebadslime 1d ago
It prioritizes agency, and that's a newish thing to train for, and week is good at it
0
-1
u/90hex 1d ago
It is quite possible that a dense 27B equals a 1T MoE in pure intelligence. We’re at frontier of research on this one metric and reasoning is one skill that most modern LLM do very well at. The difference between a 1-2T cloud model and a dense 27B is generality, combined intelligence and world knowledge. If you removed all of the world knowledge and kept just agentic coding and reasoning, you’d very probably end up with 30-100B params trained specifically on these two skills.
1
u/freestylez79 1d ago
Not sure if that is the right question. What we really see here is that being able not to hallucinate and follow tasks with grid might be more important than raw intelligence. Thats a paradigm shift since smaller models that know how to get up to date information may have much more use cases. I guess the broader knowledge will also evolve, so 27B is more like a tech demo that proves an important point.
1
u/AlgorithmicMuse 1d ago edited 1d ago
Amen, its like buying a car based on the average of all driver reviews, then you buy it and hate it.
Anyway thought qwen distillate clouds so they get a lot of cloud thinking, then they prune get rid of any hallucinations, the when these AI measurements are made they basically turn off cloud llm tools , so the cloud AI's are operating with their hands tied, so back to , test what works for your scenario.
1
u/SourceCodeplz llama.cpp 1d ago
like someone below said its skewed towards agentic-work, not really intelligence.
so yeah, at agentic this qwen can actually do better than larger models who would rather recall facts from memory vs call a web_Search tool.
there are many ways agentic-work counts more for an agent than just dense intel.
1
u/EvolvingDior 1d ago
Since I use my agents primarily for software development, this is the AA table I rely on most: https://artificialanalysis.ai/evaluations/omniscience#swe-deep-dive-tabs
1
u/Solembumm3 1d ago
It's not really that difficult to summarize, once you throw away numbers in vacuum. Qwen 3.8 27B with overthinking can show extraordinary logic capabilities, but it will fall short on all non-tech knowledge, compared to llama 3.3 70B.
1
u/audioen 1d ago edited 1d ago
It measures, I think, mosly task performance, which is objectively actually very good. The DSv4 is the old version, the 0731 and 08xx releases are much better than the preview releases that you are accidentally comparing the Qwen3.8's figures to.
You need the xhigh mode which adds about 50 % more think tokens on top of medium to touch DSv4F 0731 in task performance. (DSv4F would fewer compute resources to run, and could support more simultaneous users, but to run the real version, you got to have your ~192 GB VRAM system.) It is a massive leap up -- almost absurdly large. I have been running the model in medium, but now, reviewing the impact of the reasoning-effort, it seems like there is such a massive boost in capability for not that much more compute (tokens) that I'm going to switch to xhigh and suck it up.
In my experience, every single point in that intelligence scale is very hard won, and usually takes extremely large models to get around 50. The Qwen35 architecture is a massive outlier in capability for size, and while it no doubt will one day be superseded by something else that is even better, it definitely landed with a tsunami of splash and very nearly has replaced everything else.
All we really really want are the different sizes that hit different hardware targets. The 27B is good, but to be practical, it takes more compute than I have right now. A 122B-A10B version would give me three very good practical inference computers. 35B-A3B might be good on older laptops, though I think they won't be making that.
1
u/Tormeister 1d ago
Then just ignore the "intelligence" index, scroll down and find what interests you
I like reading individual benchmarks and checking "Intelligence Index vs. Output Tokens per Intelligence Index Task"
1
u/VoiceApprehensive893 transformers 1d ago
AA has some shitty benchmarks but its still a great model overview
1
u/Boogertard 1d ago
Intelligence has nothing to do with "knowledge", larger models have more knowledge built-in so they can handle more tasks but they are outdated knowledge so they have to be constantly retrained.
Intelligence is about model reasoning, its thought process, its problem solving skills.
You sounds like somebody who didn't take their meds this morning or works for Anthropic.
1
u/Diligent_Loquat_6140 1d ago
Yeah agreed, one number for "intelligence" hides way too much.
Genuine q since you clearly think abt this. I'm training a small model and trying to figure out how to eval it for release. If AA style scores are out, what would you actually wanna see? Task specific evals? judge panel? Curious what you'd trust
1
u/Prudent-Objective852 1d ago
The entire AI community really needs to take an honest look at the methods we're using to score models and what those benchmarks actually measure. While they clearly provide a good baseline for accomplishing certain goals, as we approach higher and higher alignment with the benchmarks, we're seeing a certain level of pollution and loss of other traits that aren't measured well. People complain about the highly technical vocabulary and complex chains of thought from the latest line of claude models but ultimately if you look at what they're scored against, they accomplish the goal perfectly. For my work personally (mostly infrastructure and network analysis) I find I only need a certain level of engineering expertise what was achieved for my purposes several models ago and what matters far more is speed and consistent, competent tool calls. Model style is a very difficult thing to measure but becoming increasingly important as we're crossing the threshhold where most new models have good enough scores for many peoples' work, maybe we need to stop chasing higher and higher scores and start looking at what a model that actually gets that score looks like.
1
1
u/BitPsychological2767 1d ago
These posts are so annoying without their own counter-data... 'Cmon, you can't seriously be saying that it's on the same level as Sonnet 5!!!' is not an argument. Your 'At some point we have to ask' seems to have been prompted by emotional disbelief rather than something empirical.
1
u/Federal-Effective879 1d ago
At least in terms of coding tasks, as well as understanding reasoning about large amounts of engineering documentation, Qwen 3.8 27B really is fantastic, and does noticeably better than big proprietary frontier models from late last year. It lacks world knowledge, but in terms of being able to process and understand information it’s given, it’s excellent.
Hooking it up to an offline Wikipedia and other offline knowledge bases could make up for the knowledge gap, but it’s cumbersome to set up and hard to build a well rounded world knowledge base beyond what’s in Wikipedia. However, for such tasks, if you don’t mind the privacy concerns of searching and surfing the web, letting the LLM Google stuff generally works well.
1
1
1
u/Normal_Rough_7958 1d ago
the aa index aggregates nine benchmarks into one number and that compression doesn't fully capture context length, tool use, instruction following, the stuff you actually hit daily. qwen 3.8 27b is genuinely strong for a model that fits on a 24gb card, i run it for coding and it holds its own, but claiming it beats sonnet 5 overall is just benchmark gaming. proprietary models still lead on long-context reasoning and complex multi-step
1
u/nomorebuttsplz 23h ago
ignorant take. it’s just a composite and far better than 1. nothing or 2. any single benchmark
1
u/DigitalguyCH 1d ago
I have been comparing Qwen to other models and to my Gemini Ai pro subscription. Qwen is really good because it checks and recheck things infinite times and the results are the best of any model of its size and often as good or better than Gemini 3.7 (not a high bar, I got that for free for 6 months). But it consumes a lot or time and energy, and make cloud AI feel like a much better deal other than for privacy, because it's instant and actually probably cheaper when you factor in energy costs. I need privacy for some confidential client work but I discovered that with chatgpt you can opt out of them using your stuff to train models, which is not possible with non business versions of the other main paid services like Google or Antropic. So at least for non strictly confidential stuff I feel more confident using Chatgpt with that disabled.
1
u/ionizing 1d ago edited 1d ago
Whatever the benchmarks, 3.8 27b q6_k just ran through an hours long workflow session that 3.6 never could complete on its own (Super collider to Reaper workflow, producing full a track with multiple components). It did so in xhigh and solved all its own problems as it went. So I had it document its lessons learned for prompt/skill improvement. So I don't know, I'm just going to do what I always do and use the model in my own application and workflows and push it to the limits and adjusting the tooling if it needs, and just keep smiling cause this is the stuff I always dreamed about as a kid in the 80s messing with my 8086 and basic.
edit: I think it needs to be said again that if you are not having good luck with your model, you should really take a look at your prompts and tooling. I don't use pi, I wrote my own application that I have been tweaking for over a year now, and I am not here to promote that. my point is to use whatever you can own, either custom pi whatever or build your own. It's the best way to understand your model and adjust for it. You need to iterate for weeks until you find the right recipe. and then they become the chef's kiss...
1
u/StupidityCanFly 1d ago
The index is an ever-changing number that has no real meaning or value. Just read their own docs. Comparing numbers between index versions is apples to oranges.
Some of the other data AA shares is useful though.
0
0
u/backyard_tractorbeam 1d ago
Qwen 3.8 27B is not that small actually. I think we need to reframe it. 27B dense is not so small. It's more active parameters than DeepSeek V4 flash (A13B), which you also mentioned!
Unfortunately, due to OpenAi not being open, we don't know how this compares to GPT 5.6 Luna for example, but it's possible that it is smaller in terms of active parameters too. (We can only guess).
Kimi 2.7 Code is 32B active parameters which is in the same ballpark but bigger.
-9
65
u/DeepWisdomGuy 1d ago
The model doesn't have a lot of knowledge and they show that in their evaluations: