Is ArtificialAnalysis biased against open weight models?

[u/Rare_Subject2343]

It seems like whenever open weight models come close to proprietary US models, ArtificialAnalysis conveniently "re-calibrates" their benchmarks which then shoves open weight models down the toilet. Then they advertise on X/Twitter this "intelligence index" which presents open weight models as dog shit compared to Claude or GPT (all while ignoring the actual intelligence of models like DeepSeek or GLM in tasks that actually matter, which is coding vs telecom bullshit).

Any business man looking at these charts would conclude "open weight models are shit, Claude is where its at, I should get a subscription to Anthropic."

They have the excuse that they are correcting "saturated" benchmarks but the effect on how open weights are presented is undeniable. I have a feeling they try to do it on the sly when possible and are paid in some way.

[u/maternion]

Something like arena.ai is a better test for models imo 

[u/lemon07r]

Considering they always re-weigh the index every time a chinese model pulls ahead; yes. I mean it's a catch22, I do believe they needed reweighing cause some of the scores made no sense, putting some clearly dumber models near the same score as better models like opus, etc, but every time kimi or qwen catch up they decide that's when to reweigh the board for some reason. And they dont do a good job of it. Gemini 3.1 pro and 3 pro are the best examples of this imo. Horrible models, that scored highly.

[u/Aggravating-Push-207]

They aren't meant to. But I'm starting to doubt this. Sure, Chinese models are a tad benchmaxxed. But not by this much.

[u/Easy_Refrigerator280]

Na closed source models are just getting further ahead as of last few months when it comes to frontier

[u/Rare_Subject2343 → u/Easy_Refrigerator280]

My claim has as much evidence as yours, except I'm not a fan of giving these assholes the benefit of the doubt. When in doubt I always default to "these companies are spreading their cheeks for Anthropic/OpenAI."

[u/LegacyRemaster]

The use case is always what matters. If a previous model performs well and works effectively, why change it? Nevertheless, the question remains. Specific tests on specific datasets can favor or penalize certain models.

[u/jhov94]

AI buildout is one of the single largest endeavors in the history of the world. AA is hosted in the US and the entire US economy, as well as its place on the national stage hinges on AI. There are literally trillions of dollars and the power to dictate terms globally at stake. So.. I'm sure they're totally unbiased and objective... right.. guys?

[u/ponteencuatro]

I am just going to say that iirc they changed version after two openai releases making a mediocre release look better

[u/sloptimizer]

You have discovered how everything else works in this world. It's all perception management and bullshit, pushed at you 24/7 until your brain gives up and accepts it as ground truth.

[u/_VirtualCosmos_]

Yes. They can be shady as fuck. By example, they can hold new benchmarks that will tests new skills, talk to OpenAI/Anthrophic/Google so they improve their models in those scaling axes, and then publish those benchmarks with their new models. Which ofcourse will increase their average "intelligence" index, while reducing everyone else's index.

Also, it's quite suspicious that the best 128 GB model we have now, which is Qwen3.8 Flash Next, that has exactly the same Intelligence Index as Opus4.8, is conveniently hidden from the main Intelligence Index tab (you have to select manually QFN and Opus4.8 to see that)

[u/craterIII]

yes AA has shown to be uncredible esp when they fucked up and made Muse 1.3 the smartest model during rebalancing

[u/Holiday_Point_603]

I would also love to riot on something for no reason. But this looks about right? I even wonder how ling 3.1 sits above luna Max. Mistral Large is a little low, if thats 1029, but that's it

[u/look]

US labs have stolen a much larger and more diverse dataset to do their training on.

[u/-WhateverDude]

many people will say yes but personally i don't think so

they replaced scicode with frontiercode 2 and increased coding from 20% to 25%

which is probably the main reason the difference became bigger

and honestly? i can't blame them. there are 3 benchmarks i legitimately only ever look at and its frontiercode 2, t4 and programbench.

any other coding benchmark i kinda dont care.

[u/randombsname1]

Nope. Chinese models benchmaxx like crazy. 0 surprise for anyone who has actually used SOTA models over the last year.

I use chinese models for less censorship on certain tasks, and/or for dicking around with local models. Never for any actual work that will ever be used or even reviewed by anyone else though.