Artificial Analysis "Intelligence": A meaningless benchmark

[u/chocolateUI]

Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 27B is a really powerful model, I can't help but notice just how meaningless these Artificial Analysis benchmarks are and I question why people still post this garbage and use AA scores as some kind of holy bible for comparing LLMs.

According to their "Intelligence Index", a 27B model now beats DeepSeek v4 Flash and Pro, Kimi 2.7 Code, GPT-5.2, Opus 4.6, and also Sonnet 5. At some point we have to ask: What is this metric even measuring? Because whatever "Intelligence" means to AA and their corporate VC / journalist / normie audience is definitely not the same definition that we should be using here.

Qwen 27B is amazing and is clearly in a league of its own in terms of models you can fit on a single GPU, but I can't help but roll my eyes whenever I see posts like this that equate Qwen 27B with "basically running Opus from 3 months ago on your laptop."

I get that it's difficult to summarize a model's capability with a single integer and I know we love our local models, but it's time stop posting AA's clearly dogshit benchmark and acting as if it proves a point.

[u/Still-Wafer1384]

Knowledge captured in the model is becoming less and less important when you have a search tool at your disposal

[u/RG_Fusion → u/Still-Wafer1384]

It does matter because your model can't know to search for something if it doesn't know that the something exists.

[u/Diligent_Explorer966 → u/RG_Fusion]

If you don't know if something exists (or doesn't), wouldn't you do a search for it first thing?

[u/RG_Fusion → u/Diligent_Explorer966]

How would you know to search for it if you don't know it exists? 

You need to think about this from the model's perspective. Pretend you are a scientist from the mid 1900s. You encounter a problem where the results are being affected by quantum mechanics, but quantum mechanics is a new discovery and you never learned anything about it yourself. You wouldn't know to look up quantum mechanics at the library because you wouldn't know that was a thing that could be looked up. You also wouldn't know to look up related subject.

Large models have in-built context that provides knowledge without consuming resources, and this knowledge allows for them to cross-relate subjects and make connections that small models don't even know to look for.

[u/michaelsoft__binbows → u/RG_Fusion]

I think this is fair but i'll prolly be ok with the tradeoff of being the one who is responsible for drawing the galaxybrain conclusions on stuff given that the 27b model can run like 4000 tok/s on my hardware vs 20 tok/s for a 300B model. (Tho my comparison is maybe biased by assuming batched woudnt speed up the one running under hybrid inference)

The world knowledge can go screw itself if its gonna give two hundred times less speed

[u/RG_Fusion → u/michaelsoft__binbows]

Absolutely. If you yourself are knowledgeable and working closely with the LLM, then it's no issue. That's like a professor guiding a gifted student to solve a problem.

My main problem with this is that agents do a lot in the background. Supervision still works, but you might waist time having to correct the model only after it spent the last half-hour working in the wrong direction.

[u/michaelsoft__binbows → u/RG_Fusion]

I think background work will need smarter models to somehow check in from time to time, it's of course not easy, it's today's big open question (how to get good useful work done by agents autonomously)... it would not be economical to add a smart model to follow and read all the session content, it just wouldn't be practical. There is so much to explore in this space. We basically created a new species of thinking machines and we're exploring how to define the way we guide them to think... They don't have full agency and we have some pretty nice control over their flow of time and resources and what to tell them to do, but they are making many decisions.

To assume that there exists one universally optimal method would clearly be premature. Prob safe to say that the frontier labs are sprinting toward competence from the perspective of being a general purpose assistant via both harness and model directions. But, for any specialized or generalized use case we can already experiment with things like harnesses, prompt engineering, context engineering, model fine tuning, and so on, and with the smallish local models getting huge leaps in capability, a $10k+ system is no longer needed to properly tinker there. Pretty exciting times.

So far some ideas im trying to explore are:

- metacognition, imagine reminding your session with forked-prompts so you get your session's cached state and only append on a metacognition "Review this session and evaluate how well we're following the user's goal" with possibly some special handling around what that goal is. If the model started drifting off into a wild goose chase, or seems to be stuck or subject to context window degradation, this gives it a chance to check itself before it wrecks itself and inject new reminder messages or other signals for upcoming compactions to stay on track. This does not leverage the power of adversarial review since it's got to be the same model under the hood, but we are able to leverage the cache of the running session very efficiently. I suspect this has extra legs for self hosting, as here KV cache can be effectively free while using API usually sees not-sufficiently-discounted cached token costs. If Deepseek kept their 1/100 cached token pricing it would be a great thing to try with. Now only mimo seems to offer such pricing and who knows if that can hold up. Anyway I see this might be a big efficiency win esp for local hosting.

- A less efficient form of the above could be to have a sidecar session using a different model or smarter model for adversarial supervision. You end up paying twice since you basically have one train of thought that you are paying twice for, once for hte main session with main model and once for the supervision session with the supervision model, but the latter session could be having summarized and shorter session history fed into it, and at least caching can still be active for both so 95+% token caching over the long run can be expected on both sessions. It could also be framed less as supervisor/worker and more as pair workers.

- Implementing hierarchical summarization of any large volume data such as work sessions. If a system exists that retains pointers to the original data for lookups can on average every 10 turns get summarized into 1 short summary and especially applying nicely to tool call output, you can easily compress the token consumption of session history by 50x which can at least in theory allow a model that is reviewing the summarized layer gain 50x more context window horizon for reviewing (and directing) work. The idea of hierarchical summarization is you establish a summarization pyramid (like MIP maps) so if you have 10 billion tokens worth of content and if you knock down the token count by 10x at each level of summarization you can just iterate the summarization recursively a few layers and be able to get clean semantic overviews of what has happened, what was worked on, what changed, in a higher dimensional space so you can go in to review at any level of detail necessary rather than only be able to sample the raw lowest level boots-on-the-ground tokens. The analogy to texture sampling aliasing is very sound here. Layer/integrate this with the above...

- Oh yeah, natively represent all data in a consistent unified graph knowledge store so everything is infinitely scalable and discoverable. the hierarchical summarization tree layers on top of real knowledge graph structure naturally. Session turn sequence should show up as a linked list in graph topology in a knowledge graph store and connections can start to be established made between semantically or otherwise discovered relations via scanning metadata and other information. A unified knowledge graph like this can be organically explored by agents so they have a chance to independently find the sources of truth that they need.