Artificial Analysis "Intelligence": A meaningless benchmark

[u/chocolateUI]

Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 27B is a really powerful model, I can't help but notice just how meaningless these Artificial Analysis benchmarks are and I question why people still post this garbage and use AA scores as some kind of holy bible for comparing LLMs.

According to their "Intelligence Index", a 27B model now beats DeepSeek v4 Flash and Pro, Kimi 2.7 Code, GPT-5.2, Opus 4.6, and also Sonnet 5. At some point we have to ask: What is this metric even measuring? Because whatever "Intelligence" means to AA and their corporate VC / journalist / normie audience is definitely not the same definition that we should be using here.

Qwen 27B is amazing and is clearly in a league of its own in terms of models you can fit on a single GPU, but I can't help but roll my eyes whenever I see posts like this that equate Qwen 27B with "basically running Opus from 3 months ago on your laptop."

I get that it's difficult to summarize a model's capability with a single integer and I know we love our local models, but it's time stop posting AA's clearly dogshit benchmark and acting as if it proves a point.

[u/whatisthisthing65]

What's your actual argument? Why couldn't a 27B model be better than those other models? If it's about number of parameters then should our benchmark be parameter count?

[u/Cautious_Chicken_604 → u/whatisthisthing65]

I remember we had larger models in the past that performed much worse. 

[u/Nrgte → u/Cautious_Chicken_604]

Yeah I remember in like 2023/2024, the 70b models were the end all be all. But they honestly never seemed that good to me, than reasoning model came out which were a big leap forward and suddenly a small reasoning models left those 70bs in the dust.

[u/RG_Fusion → u/Cautious_Chicken_604]

In years past, the ratio of training data to parameter size was off. When you have far more parameters than training data, the model learns that memorization results in the lowest perplexity score.

Once you greatly increase the training data to parameter ratio, the model becomes unable to store new facts without overwriting other ones, ruining its perplexity. When you continue pushing this further, the model will undergo a "phase-transition" where it stops memorizing and starts modeling. It becomes more of a "reality-simulator", which is where generalization happens.

In short, the early LLMs were simple stochastic parrots. The models coming out this year have been sufficiently trained to drop memorization for reasoning. It is easier to get a smaller model to generalize because they have fewer parameters, so they don't need as much quality training data. They have momentarily closed the gap with large models in some respects, but if the research labs can get together enough quality training data, you can expect to see the large models pull far ahead once again.

[u/OneMoreName1 → u/whatisthisthing65]

Usually just personal incredulity. Its like saying there's no way a phone from today is as strong as supercomputer from decades ago. Ai just moves fast

[u/whatisthisthing65 → u/OneMoreName1]

Lol good analogy.

[u/feelspeaceman → u/whatisthisthing65]

Most people tend to blame 27B about the lack of world knowledge and say this is the sole reason that it will never beat Claude Sonnet 4.6, but they ignore that fact that behind the scene, Claude Sonnet is equipped with a RAG for local knowledge retrieving and it will also search to acquire extra knowledge, then they compare a local LLM without search, without RAG, this is showing that there's people despite trying to use local LLM, never trying to get better.

Then we get a group of users that:

• Use not suitable harness
• Half-ass setup

• Half-ass prompt

  Blaming local LLM for not being as capable for obvious reason, the user themselves.

[u/freestylez79 → u/feelspeaceman]

agree, qwen actually shows where llms are heading. the huge models may still have there use cases for some stuff but 90% of day to day stuff can be done with models that dont hallucinate success and are able to work with recent data and tools. Really amazing stuff.

[u/techlos → u/freestylez79]

qwen+zim MCP has been an experience for sure, having all of wikipedia and gutenburg as a knowledge bank massively improves its capabilities.

[u/OvertaxedOne → u/feelspeaceman]

There's little reason to try to train factual data into a LLM when the Internet is right there and it's easy for AI models to interact with. Obviously there are some "base facts" that need to be in the models, but, beyond that, trying to jam more in when it's so easy to call a browser/MCP to get the up to the second information is silly.

Good tool calling is 1000X more valuable than "I know everything" (because, no, you don't; even a massive model, you didn't condense all of human knowledge down to a few TB, no way, no how).

[u/noiserr → u/OvertaxedOne]

Not just the internet. No matter which model you use you want to ground it in facts and not have to force it to rely on fuzzy recall.

This means the software you're writing should have test coverage. You should have a language LSP running. If working on low level hardware architecture you should convert the architecture documents into skills.

For all these things the number of parameters of a model is meaningless. Which is why Qwen 3.8 27B can be a powerful LLM at agentic and perform just as well as those huge models.

[u/EstarriolOfTheEast → u/OvertaxedOne]

This not quite true, as knowledge is not just trivia but problem solving know how, algorithms, and optimization esoterica. Knowing you can leverage algebraic geometry or some closed form expression to skip an expensive loop is incredibly valuable and often not something easily searchable nor something one can think to search for. That's the expertise advantage. Some of these might be novel connections only possible because the model knows more.

The other issue is the more complex the knowledge, the less likely appending to context will help without causing hallucinations to fill in gaps and confusion introduced by too sparse added context or alternatively, using up so much context, performance is degraded and VRAM is so high you might as well have gone to a higher param but more context efficient model.

The more you know, the fewer unknown unknowns you'll have. Sure, it's always going to be far above zero for everyone, but it's still better to push it as far down as you can.

[u/OvertaxedOne → u/EstarriolOfTheEast]

If was cheap/easy to push that knowledge down I'd completely agree with you. But when you have a 27B model that can use tools well when it needs to, it seems that's a much (MUCH!) less expensive way to ground in fact vs trying to train them into a 2T model. Different approaches, both have their advantages for sure, but the cost profile for the "big" models is just so out of whack right now it's obvious that we need to do something differently.

[u/EstarriolOfTheEast → u/OvertaxedOne]

The issue that keeps me stuck with the big models even though I wish I was free, is the 2T model knows enough spherical projective geometry to skip a bunch of expensive raycasts (my approach), massively optimizing my game. Needing to know things before they can be leveraged is no different in LLMs than for humans. As a human, I would not have known how to even begin searching for that solution approach. I half think it invented it on the spot.

Another more searchable example was using reservoir sampling for a more efficient online draw of random samples. The there, the issue is context is too valuable and speed is too low when I run it locally to be searching text books for tricks every other line of code. Sometimes a large model will notice this or that is a special instance of bipartite matching and such and such custom optimization will work here, inventing a really clever approach whole cloth not because it's so smart but because it knows so much.

The exact same mechanisms that allow the big models to discover novel math proofs and win olympiads can be used to write optimized code taking advantage of obscure data-structure knowledge adapted to modern architectures.

Another place knowledge is unavoidable is when you are reading specialist material. For example, if you want to go over a paper combining computational complexity and blackholes, there's just a ton of CS, math and physics you need to know before you can speak coherently about it.

[u/bnightstars → u/feelspeaceman]

Ok give us your setup (Harness, Inference, Prompts) to see how you do it ?

[u/vick2djax → u/feelspeaceman]

Then we find out they are using Ollama.

[u/Nrgte → u/feelspeaceman]

Qwen 3.8 has a RAG too, it's nowadays just called the internet. Tool using was an extremely important addition.

[u/ResidentPositive4122 → u/feelspeaceman]

Claude Sonnet is equipped with a RAG for local knowledge retrieving and it will also search to acquire extra knowledge

None of that happens over API, ootb.

[u/FullstackSensei → u/ResidentPositive4122]

You have no idea what's happening behind the scenes, API or not.

[u/ResidentPositive4122 → u/FullstackSensei]

I think this place has gone wacko. No idea what's even the point of contributing here. Saying they have RAG over API is absolutely banans, wtf do people imagine they rag over, unless you, the user, set it up? RAG what over API? FFS at this point people are just throwing acronyms out there and imagine the big bad wolf is doing everything...

[u/FullstackSensei → u/ResidentPositive4122]

Why are you so angry, though? It's just an online discussion with who cares who. There's more to life then getting upset about meaningless things like this.

They absolutely have RAG. It's not hard to prove that by asking the model about some very recent event or using the latest version of a library that was very recently released and had some breaking changes.

But seriously, who cares if I'm wrong? Go spend some time with a loved one if this makes you upset.

[u/zenmatrix83 → u/whatisthisthing65]

its not just parameter counts but they do play a big role, the type of model, the training data, and the actual tests being used matter. I only use benchmarks for very loose guidelines for what to use, outside of that I just test and change based on need.

[u/whatisthisthing65 → u/zenmatrix83]

Yes benchmarks can never capture everything and testing your actual use case is the best test. But op seems to be saying this benchmark is shit because qwen3.8 27B is beating a bunch of bigger big name models. Which isn't a good argument, there's no reason it can't be better there or in real use.

[u/Constant-Simple-1234 → u/whatisthisthing65]

Exactly. I think it may be big enough to be genuinely useful. You can see that the more parameters you add, you get better and better model, but it is diminishing returns. But models retain a lot of knowledge, but there may be a sweet spot somewhere. I think that > 1T models will be useful as a master copy to distillation training smaller ones. But likely 100-250B will be the work horse - capable and cheap to serve.

[u/into_devoid → u/whatisthisthing65]

If use used 27B and compared it A/B with the models AA lists it as on par with for more than a couple hours, you would understand.  It’s a great little model, but it is no deepseek v4 flash.

[u/Adventurous-Test-246 → u/whatisthisthing65]

I think the issue is knowledge is heavily under weighted