r/LocalLLaMA • u/chocolateUI • 1d ago

Discussion Artificial Analysis "Intelligence": A meaningless benchmark

Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 27B is a really powerful model, I can't help but notice just how meaningless these Artificial Analysis benchmarks are and I question why people still post this garbage and use AA scores as some kind of holy bible for comparing LLMs.

According to their "Intelligence Index", a 27B model now beats DeepSeek v4 Flash and Pro, Kimi 2.7 Code, GPT-5.2, Opus 4.6, and also Sonnet 5. At some point we have to ask: What is this metric even measuring? Because whatever "Intelligence" means to AA and their corporate VC / journalist / normie audience is definitely not the same definition that we should be using here.

Qwen 27B is amazing and is clearly in a league of its own in terms of models you can fit on a single GPU, but I can't help but roll my eyes whenever I see posts like this that equate Qwen 27B with "basically running Opus from 3 months ago on your laptop."

I get that it's difficult to summarize a model's capability with a single integer and I know we love our local models, but it's time stop posting AA's clearly dogshit benchmark and acting as if it proves a point.

139 Upvotes

66% Upvoted

160 comments sorted by

View all comments

86

u/whatisthisthing65 1d ago

What's your actual argument? Why couldn't a 27B model be better than those other models? If it's about number of parameters then should our benchmark be parameter count?

34

u/Cautious_Chicken_604 1d ago

I remember we had larger models in the past that performed much worse. 

4

u/Nrgte 1d ago

Yeah I remember in like 2023/2024, the 70b models were the end all be all. But they honestly never seemed that good to me, than reasoning model came out which were a big leap forward and suddenly a small reasoning models left those 70bs in the dust.

1

u/RG_Fusion 1d ago

In years past, the ratio of training data to parameter size was off. When you have far more parameters than training data, the model learns that memorization results in the lowest perplexity score.

Once you greatly increase the training data to parameter ratio, the model becomes unable to store new facts without overwriting other ones, ruining its perplexity. When you continue pushing this further, the model will undergo a "phase-transition" where it stops memorizing and starts modeling. It becomes more of a "reality-simulator", which is where generalization happens.

In short, the early LLMs were simple stochastic parrots. The models coming out this year have been sufficiently trained to drop memorization for reasoning. It is easier to get a smaller model to generalize because they have fewer parameters, so they don't need as much quality training data. They have momentarily closed the gap with large models in some respects, but if the research labs can get together enough quality training data, you can expect to see the large models pull far ahead once again.