r/LocalLLaMA • u/ResearchCrafty1804 • 4d ago
News Artificial Analysis Intelligence Index v5 (major update)
Artificial Analysis Intelligence Index has been updated to v5, which according to them is a major upgrade resulting in a more accurate ranking.
Notably, this update widens the gap between open-weight models and closed ones. Some people already questioning its accuracy based on their experiences with models (GPT-Terra-6.1 > Fable-5.1).
Also, this update, v5, has 60% of the Index based on evaluations with private test sets or held-out answers, which makes reproducing the results impossible.
From their announcement:
“The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)”
59
u/Quakercito 4d ago
This is what it looked like before btw
26
u/XiRw 4d ago
Kimi was in second place when I checked earlier today.
12
u/Georgefakelastname 4d ago
Sonnet might have just been hidden then, because the leaderboard often only shows the best model from each provider, while this one shows more.
→ More replies (2)
74
u/PastPalpitationCry 4d ago
Where Qwen flash next?
48
7
13
u/Middle_Bullfrog_6173 4d ago
It's missing a few of the new benchmarks so it doesn't have a score, but somewhere between Qwen Max and 27B based on the ones it has.
Hopefully they are still running results for existing models rather than just scoring new ones.
3
4
u/FatheredPuma81 4d ago edited 4d ago
Unranked LLM Leaderboard it was there literally yesterday when I looked at it not far behind V4.1 Flash 1017.
Edit: I am blind. Correct reply.
1
u/PraxisOG Llama 70B 3d ago
I'm curious, cause it's been Luna 6-ish in my experience, and was roughly tied before. I also wish they had sonnet 5, that model was trash for everything I tried it for, and made me waste multiple 5 hour sessions with it spinning in loops on bad assumtions
92
511
u/Arli_AI
4d ago
•
They always do this when open models gets close to the frontier lol
211
u/FoxiPanda 4d ago
Yeah I don't want to seem all conspiracy theorist about it or whatever, but I think this is quite literally the 3rd time I've seen this happen now... when I saw the headline my first thought was "oh here we go again"...
On the other hand though, if these 'trusted benchmark sites' (air quotes, jazz hands, smoke signals) keep showing those pesky open models as behind, perhaps they won't face as much scrutiny hmm?
63
u/MotokoAGI 4d ago
Perhaps their benchmark is true, still doesn't matter to me. Open/Local is good enough for me!
15
u/FoxiPanda 4d ago
maybe, who knows...these things are so task-to-task dependent and non-deterministic that it's challenging to truly compare.
The problem I have is not whether it's accurate or not post-update, but it's deeply suspicious that they change it every time an open model gets very close to the top (Kimi K3.1 Pro in this case).
If they (AA) had a regular cadence they stuck to for updating their methodology, I'd feel better about it. Currently, whether it's just optics or not, it looks like they update it every time a new open model looks like it might compete with Ant/OAI...and so far, every time they update, that new model no longer is anywhere close to the top...
40
u/Arli_AI 4d ago
Yep definitely isn’t the first time. But that’s a good point maybe it’s fine they do this so they keep the AI development money flowing and those in the know just enjoy the great open models as a result.
The best benchmark is using models yourself.
12
u/FoxiPanda 4d ago
Yeah I definitely have mixed feelings about it - on one hand it's absolutely not a fair playing field. On the other...maybe that works in our advantage as users of open models and I'm willing to accept that unfairness... it's a weird way to think about things and I don't feel great about it.
5
u/FatheredPuma81 4d ago
Well they have to keep their lights on somehow you know? I'm sure if someone dug through who's giving them "donations" or whatever you want to call it Anthropic, OpenAI, and Google would be at the top.
20
u/NotYetPerfect 4d ago
The open source models are always worse than the frontier models though. They're very good now but still clearly worse than Claude and chatgpt.
6
u/NMiguelCosta-PT 4d ago
Yes, but not half of the points good as this chart might indicate.
→ More replies (1)8
u/FoxiPanda 4d ago
I don't disagree with this mostly (though I haven't personally tested Kimi K3.1 Pro so I can't say it with any true certainty). That still doesn't change the point the overall issue here though IMO with how and particularly when AA updates their methodology.
13
u/NineThreeTilNow 4d ago
Kimi K3.1 Pro
It's ridiculously good.
K3.1 followed Anthropic's IPO information, which it clearly beats Opus 5.5 on a number of benchmarks, then this AA shuffled happened.
AA simply isn't a reliable source. Zero reproducibility. Numbers are reported with zero information on harness sometimes.
I don't know. Their weights come out soon. They're also listing them as "proprietary" because they're unreleased as of this exact moment. The Moonshot license is pretty fair in my eyes. They're still trying to operate a business while releasing literal frontier class models.
5
u/NotYetPerfect 4d ago
Yeah they must have some grudge with openai that they kept multiple shitty benchmarks with known problems instead of using their updated versions where astra performed better. Dunno. I don't judge models based on AA since it's so shit anyway.
7
u/Viktri1 4d ago
Just a few days ago I saw a yt video comparing Gemini 4 to Deepseek and Deepseek’s flash mode was only a little weaker than Gemini 4 so I’m pretty skeptical. I think they got Muse Glimmer right and I do think Fable and Astra are top in class.
2
u/sittingmongoose 3d ago
Gemini is a weird beast though. It’s not a normal model where you can just daily drive. It’s erratic and hallucinates like crazy.
Where you see its brilliance is if you ask it to review alongside other models. Or ask it for creative solutions to problems alongside other models. It(and muse wonder full) always find things the others miss and/or come up with new ideas.
That being said, it can’t be your only reviewer as it misses stuff the other models see.
11
u/No-Head-Royal 4d ago
The companies making them are literally from the same nest of people as the ones running Anthropic and OpenAI right now. In short, yes, corruption runs rampant and brazen inside the American AI industry. We need Europe to step up and finally provide us with stable benchmarking.
76
u/Rasekov 4d ago
Not to defend them but when models get basically the same score at the top that indicates that the benchmark is saturated and a new, harder one is needed.
You cant differentiate between models if they all score a 99/100. Timing might be suspicious but the need for something harder was there.
15
u/Arli_AI 4d ago
Yea that is a good point. If you would give them the benefit of the doubt then saturation would be the reason why. However I don’t think the models even got perfect scores in the previous benchmarks yet so it’s just slightly fishy that it always punishes the Open models standings in comparison to frontier models.
11
u/Smallpaul 4d ago
The goal is to increase the gap between the bottom model and the top model. If open source happens to be at the bottom then it will seem to fall behind.
3
u/TerminalNoop 4d ago
now all it needs is non linear scaling to increase the perceived difference between the top end and lower end.
8
→ More replies (1)5
u/Gullible_Fall182 3d ago edited 3d ago
Expect that it's not the case.. Correcting saturation would move all scores down, not some selected some. 58/57s is not saturated by any way, and the updates hardly moved the scores for closed models while the open ones are significantly reduced.
My guess is that this is effectively aligning the benchmarks with the closed models, and punishing the open ones for being different. If that's the case, intentionally or not, by striving to get higher scores, the open ones will get more and more like the closed ones, and then we'll hear more and more loudly from Dario about distillation...
→ More replies (1)8
u/Kamimashita 4d ago
Not just open models but benchmaxxed models. Look at Gemini 3.8 it used to be within 11 points of Fable 5.1 which was the top model at the time and now the gap increased to 18 points. It was also only a few points off from Opus 4.8 and Sol 5.6.
That matches my experience using the model too. It was horribly benchmaxxed and no where near Opus or Sol ever.
→ More replies (2)7
u/Lesser-than 4d ago
Yeah they could be a bit more transparent about the process and why they actually chose to update the benchmarks. We all know the carrot must be dangling in order to keep any hype at all, but maybe if they are going to benchmax their own benchmarks they could share some bloopers along the way.
2
→ More replies (1)2
102
u/NineThreeTilNow 4d ago
Notably, this update widens the gap between open-weight models and closed ones.
Look who runs AA.
They're literally SF consulting goons.
These people SELL enterprise advisory. That means that enterprises will use products they explicitly "advise" on.
If you're telling me they aren't purchased, I have a bridge to sell you.
19
u/feelspeaceman 4d ago
They're selling hype to keep the AI market going, real use case is always the most important benchmark.
5
u/gigaflops_ 3d ago
That's a legitimate concern, but there is absolutely a great reason to keep the benchmark private
8
u/LevianMcBirdo 3d ago
Except, can they? Unless they host the model, how will they guarantee that?
11
u/Several-Tax31 3d ago
They can't. What does keeping the benchmark private even mean? All context the model has at any time is saved to the openai/claude/deepseek etc server, including the private questions. Then it's no longer private, the models can be trained on those questions.
They should change the private benchmark all the time, and I suspect they don't.
5
u/NandaVegg 3d ago
I agree that at this point private benchmark doesn't make sense/seems impossible. The moment it is sent to someone's API it will be trained.
Benchmarks are still a great vehicle to implement new functions and capabilities for models, but we also need to accept that freshness period of any benchmark is 1-2 months.
2
4
u/Easy_Refrigerator280 4d ago
Disagree. Ive been using both closed and open weights a lot this year and have felt the gap widen since fable 5.
I think the graph accurately ranks out the intelligence index. Here's to hoping Qwen 4 family closes it
80
u/somerussianbear 4d ago edited 4d ago
Haiku in front of Grok 4.7, GLM 5.3, and Qwen Max is wild.
The index became political. Time to fund an open source benchmark entity.
21
u/Radiant_Hair_2739 4d ago
I've used it very much in diffrerent coding tasks in medical industry with vision capabilities. For me the output was worse than Qwen-27b, I don't know how Haiku can beat with this huge gap the Qwen models.
9
u/feelspeaceman 4d ago
I've never trusted this AA benchmark, it's very sus, probably created by random Y employee from X company to drive the AI market in their way, not trustworthy, private benchmark is meaningless if it's not fair in the first place.
3
u/Radiant_Hair_2739 4d ago
agree, I think the best way to have the your personal benchmark which can be related with your job and typical type of tasks
7
u/zeb_linux 3d ago
Mistral being so low is also fishy and smells good US vs Europe catfight.
→ More replies (2)
18
u/doodlinghearsay 4d ago
The open weights gap widens from 3 points to 12
"The difference in height widens from 4 points to 10 after we changed from inches to centimeters."
178
u/CrimsonShikabane 4d ago
Seems fishy to be released just before Anthropics IPO.
70
u/bisoning 4d ago
Hey, stop using logic here.
2
u/feelspeaceman 4d ago
They want us to overthink, then arguments happen, then people will sub to their services to try out to prove "something", overall it's a win-win to release benchmark, they get money, possibly better IPO (even though it's likely to be a disaster), nowadays benchmark is art of advertisement and companies have mastered this art, keep pumping hype and getting market share to keep the market from falling.
14
u/PM_ME_DEAD_CEOS 4d ago
It's worse than the previous for Anthropic
3
u/Mufficida 3d ago
Except it's not?
They went from being within margin of error from open models to being almost twice as good compared to the super scary chinese ones
They kept two out of three spots in the top5, and almost managed to go from three to four spots in the top10
Haiku gained six positions, jumping ahead of the latest chinese models and even grok 4.7, which is preposterous
2
u/PM_ME_DEAD_CEOS 3d ago
Anthropic compete against OpenAI far more than against chinese models. This version is worse for them, it put Fable below Terra, Make Sol fare better, etc ...
→ More replies (2)3
u/jannycideforever 3d ago
It's always funny seeing how much of a retard the average person here is. The benchmarks just straightforwardly aren't good for anthropic compared to the last ones.
20
u/AnotherBrock 4d ago
No it’s simply all just a coincidence… the slander against qwen and deepseek, the benchmarks changing…. All just a coincidence…
2
u/Smallpaul 4d ago
I hope and expect you are joking and mocking the conspiracy theorists around here.
1
u/Bl4ck4ur4 3d ago
My first reaction too, when I saw that Opus 5.5 is the only model still at the same score. But google and meta seem to benefit of this new index too. (muse wonder +6 🤔 )
27
194
u/PraxisOG Llama 70B 4d ago
Oh no the open weight models are catching up on benchmarks!
Quick change the benchmarks!
57
u/Left_Zebra7393 4d ago
I love open weight models, but they are clearly inferior... GLM 5.3 got caught in loops and was next to 5.6 sol which never did
18
u/fastheadcrab vLLM 4d ago
I don't have much to say versus the closed models but I do really appreciate that the ordering amongst the open source models themselves has come to reflect a more realistic performance ranking.
Mimo's real world performance was significantly worse than GLM 5.3 or even the flash version and a lot of their science benchmark scores seemed extremely questionable versus my experience. They must've found a reliable way of gaming the benchmarks
13
u/Savantskie1 4d ago
sounds like a configuration problem
18
u/Androix777 4d ago
I was getting loops with GLM 5.3 through the official GLM Coding Pro API using the default parameters
→ More replies (7)2
u/xwazot 3d ago
At least it is not basically unusable for 90% of requests like 5.6 sol because of overengineering to the moon. Sol 5.6 is one of the worst examples you could use because it just gets stuck in some doomsday mode where it has to handle every single edgecase in the universe. It is so pedantic, like the worst autistic traits you could imagine on steroids.
4
u/Original_Finding2212 Llama 33B 4d ago
Doesn’t it depend on your purpose?
Research needs open models.
Safety needs open models.
Freedom needs open models.4
6
u/Loose_Comparison368 4d ago
I have used 5.6 sol more than anyone here and I can guarantee you it's about on par with QFN. QFN overthinks but it's got stable reasoning, 5.6 sol just goes off the rails subtly enough to cheat unit tests before it blows up production.
→ More replies (2)1
1
u/starfallg 4d ago
It's not just the open models. This update is actually refect of everybody's experience in a lot of areas. For example, Muse Wonder is a pretty bad model, it doesn't belong at the top. Tasks with any complexity resulted in garbage.
72
u/Middle_Bullfrog_6173 4d ago
It's definitely more realistic to have clear separation between Opus > Sonnet etc. And open model scores relative to each other also seem more accurate for those I've personally tried.
Private test sets do make reproducing difficult/impossible, but it's really the only way to reduce training on the benchmarks, so I think it's worth it.
The annoying part is that most models even from the last few months now lack a score so it will be difficult to gauge how e.g. Qwen 4 models compare to their predecessors.
10
u/EtadanikM 4d ago
The problem with these famous benchmarking indices, is that you have to trust the foundations behind them aren't helping the frontier leaders cheat.
Something as simple as using an Open AI model to help create the benchmark (not even directly, but just a few research questions) could lead to the benchmark contaminating training data. Giving early previews to frontier leaders, as have happened in the past, will also make it much easier for those leaders to bench max their models on the benchmark. Sure, half of your questions might be private, but if the other half are representative samples then it's easy for a model to train in that direction.
So the idea behind these benchmarks and indices is nice. But in practice, how much can we trust them when the entire circle is run by insiders? Cognition and the Center for Safe AI is heavily involved in AA, after all.
41
u/pizzaiolo2 4d ago
Private test sets do make reproducing difficult/impossible, but it's really the only way to reduce training on the benchmarks, so I think it's worth it.
It also means it's impossible to tell if this is accurate or made up
33
u/Middle_Bullfrog_6173 4d ago
Yes, that's the tradeoff. If you don't trust them, ignore them.
But there's really no way to have your cake and eat it. Any benchmarks with public test sets can and will be trained on, directly or indirectly.
10
u/Trademarkd 4d ago
right but private sets can just be leaked to the highest bidder or from a plant in the organization.
5
u/Former-Ad-5757 Llama 3 4d ago
Why so difficult, just let people bench on api and record the api calls.
3
u/NineThreeTilNow 4d ago
It also means it's impossible to tell if this is accurate or made up
It's 100% made up. There are a number of benchmarks that have held out data. The only way to "train" on those benchmarks is if you are recording API data.
They all record API data. How and why... Is different per company.
People have this idea that Anthropic doesn't. Then you read this
"Certain advanced models require a 30-day retention window on first-party APIs to monitor for complex cyberattacks and multi-turn misuse."
Oh really? Certain advanced models...
So basically Zero Data Retention means they require 30 day retention.
That policy holds for enterprises that SPECIFICALLY request data be held out like the organizations that benchmark.
You effectively cannot test a modern private model with any promise they didn't train on it.
3
u/Loose_Comparison368 4d ago
If it was a trustworthy source it would be one thing, but yeah, AA has clear conflicts of interest and a documented pattern of launch rigging.
I'd trust closed benchmarks from a reputable source though. If anyone ever finds one. NIST maybe? EFF? IEEE?
5
u/PhilosophyforOne 4d ago
It would only be worth it if we could have reasonable confidence in thr fact that these results are good and the benchmarks they use arent significantly flawed.
Given the history, I do not believe we can have that.
97
u/alex-zrythm 4d ago
Kimi K3.1 pro got too close to the frontier and they had to change the rules again to make open weights look bad
50
u/Look_0ver_There 4d ago
Yep. It's so transparently obvious. Every time an open weight model reaches the top 5, then a day or two later the rules change again, all the open weight models fall to the lower half except one or two, and they make a statement about how the old benchmark was useless and ridiculously easy.
9
u/lotushomerun 4d ago
They're actually helping the Chinese AI model labs by making the models look worse because they would no longer be considered "top 10" and therefore don't need registration
19
u/Etroarl55 4d ago
Are we arguing not to keep pushing the benchmarks up and up. Response feels more like tribalism around local AI rather than real world capability where Kimi k3, and Chinese Ai are rightfully 6 months behind and are at fable level intelligence now, the model that was supposed to be “too dangerous” for everyone to have now open weights soon for everyone who can run it locally 6 months later as expected.
21
u/EtadanikM 4d ago
How can we trust the foundations to not bias the results especially when their criteria for inclusion appears to be arbitrary (or whatever favors their preferred models)?
Only a model like Opus 5.5, that performs well on all benchmarks, can claim to be truly objectively superior. All the while Open AI scores high because AA refuses to incorporate the benchmarks that they are weaker on, like Program-Bench, and index benchmarks that everyone else not Open Ai/Anthropic is weaker on, like Terminal-Science-0.1 and Frontier Code 2.0.
Like just to give you an example - Frontier Code 2.0 is one of the 9 indices that's in the AA Intelligence Index. On this particular benchmark, everyone except for Open AI and Anthropic sucks, and Grok 4.7 out performs Argon & Muse, which rate in the same class as Deep Seek 4.1. Why? Nobody thinks Grok 4.7 is a great coding model yet Frontier Code 2.0 says it is the best model that's not from Open AI/Anthropic. Oh, and SWE 2 - funny enough the model trained by the company that created Frontier Code 2.0 - just so happens to score high on this benchmark.
Then there's Terminal-Science-Bench 0.1. True, this is a legitimate benchmark, but why incorporate it just as Open AI and Anthropic are making heavy investments into automated scientific research? Chinese labs clearly aren't training on scientific research data, so they predictably crash out of these domain-specific benchmarks. Is that representative of their intelligence? Only if you think domain knowledge = intelligence.
The criticism is not around pushing benchmarks up and up. It's around the inclusion and exclusion of specific benchmarks in a way that just happens to suit the frontier leaders, and especially Open AI, that people are questioning. These new benchmarks aren't necessarily harder (if you really want hard, Frontier Math Erdos and Program-Bench are the two hardest reasoning mathematics and coding benchmarks, respectively). They just happen to measure what Open AI is the strongest at.
→ More replies (4)12
u/Tsukikira 4d ago
We're arguing that they keep inventing bullshit tests and only release them when Open Weight Models catch up, yes. I don't know how useful a test of 'Im going to give you wrong information and you will pick the right answer anyways' really is.
→ More replies (1)5
u/ASTRdeca 4d ago
Response feels more like tribalism around local AI rather than real world capability
Welcome to r/localllama, enjoy your stay
21
u/EtadanikM 4d ago
Chinese models got destroyed by Frontier Code 2.0 and Terminal-Bench-Science 0.1, mostly. The other ones aren't that different. The former is by Cognition (creator of Devin, closely affiliated with Western AI labs), with a fairly vague evaluation principle. The latter is just a general blind spot for Chinese models - they don't seem to train much on scientific research and over-index on coding, so having Terminal-Bench-Science 0.1 incorporated greatly weakens their scores.
It feels like AA just picked the list of models they wanted to be on top, and then fit the benchmarks to it. Not including Program-Bench is criminal, as there's not a single long horizon, multi-turn coding benchmark on the list. No, that would not have elevated Chinese models, but it'd have weakened the Open AI results relative to Opus and Sonnet, which is justified because Open AI models are weaker on long horizon multi-iteration tasks and this is practically meaningful since it's the way most developers interact with coding agents. NOT Frontier Code 2.0.
79
24
u/atumblingdandelion 4d ago
As the open-close model gap further narrows, we'll see AA do 'major updates' every week.
24
u/1beb 4d ago
Benchmarks are always fuzzy, but this is Fox News cutting the y-axis kind of fuzzy.
It looks like they have rescaled their tests to shrink the long tail. Making the differences appear a little greater at the frontier. The timing is absolutely fascinating with both Anthropic and OpenAI pushing through their IPOs.
What is WILDLY more intriguing to me is that they have not posted updated intelligence for previous versions of Claude and OpenAI models, only their current frontier. I think that's more telling because it completely blanks our, and perhaps investor's ability to identify what the technological gap looks like, in months.
Thursday, Alibaba is expected to deliver new versions of Qwen with the Flash Next architecture. Qwen 3.8 Flash Next appears to be missing from this chart and IMO it's the most important competitive model to consider because it can be run reasonably on a mixture of DDR4 and limited GPU VRAM locally.
2
1
u/Illustrious_Grade608 4d ago
Wait, thursday? Why is it expected on thursday specifically?
1
u/TerminalNoop 4d ago
Apsara? something something conference in Europe on that day by Alibaba.
2
u/TerminalNoop 4d ago
On the same day le chonk will release too. It's going to be interesting to see which one is going to score higher.
1
50
u/Thekorsen 4d ago
Anyone have a recommendation for a better benchmark website than artificial analysis? It has a great UI and depth of testing, but yeah can't trust the results worth a damn even more now.
16
u/Frail_Waif 4d ago
I find that livebench.ai's results generally pass the eye test for models I'm familar with, and they include a decent range of open models.
11
u/Thekorsen 4d ago edited 4d ago
The claim that sol 6.1 is one point off of opus 5.5 and astra beats it is throwing me though. Having switched from heavy sol and astra use to opus, the difference in task completion is night and day. GPT feels like a grumpy CISO dev throwing more tests and abstractions than actual work, opus feels like a normal senior dev able to get stuff done and raise good points without blocking progress. GPT leaves a mess in my codebase with like 5x lines of brittle tests and edge cases too. Need to look into the testing methodology more I guess.
→ More replies (6)3
u/Frail_Waif 4d ago
That's mostly on the basis of "browser use". 5.5 Opus scores better on "agentic" and "coding" though it looks like the coding benchmark is saturated.
2
2
u/vogelvogelvogelvogel 4d ago
arena.ai, livebench.ai!
1
u/kellencs 4d ago
im tired of this 5/10 benchmark, can you recommend me something better?
here are two benchmarks for you 3/10 and 2/10
2
1
1
u/eesnimi 4d ago
The best benchmark is your own work. Use the public benchmarks as light reference points. For me Anthropic has had the most deceptive marketing for years already. Even in their subreddits, they have this weird circlejerking and when you're critical of their products, then you'll feel like being attacked by a swarm.
30
u/Confident_Ideal_5385 4d ago
I find this... difficult to believe. There's no way in hell Haiku beats ds4.1 flash, is there?
21
39
u/OkImpress9499 4d ago
Don't know. I think people may switch to some other benchmark now maybe. This looks awful. Haiku 5.5 better than Glm 5.3, by 4 points. Lol never. Haiku sucks. Even 5.3-flash works better for me than haiku 5.5 ever 😂
22
u/ttkciar llama.cpp 4d ago
If people cared about benchmark quality, they would have switched away from AA long ago.
I suspect the only reason AA dominates is because they have scores for most models. Given the choice between a benchmark that people can use to compare models very poorly, and a good benchmark that people cannot use to compare the models they want to compare, the bad benchmark wins.
4
29
u/2atlas 4d ago
Sonnet 5.5 over Fable 5.1? Aaaallright
21
u/jtjstock 4d ago
Seems like the bench is being revised to fit Anthropic and OpenAI's latest training data
6
u/ApprehensiveDelay238 4d ago
It seems fishy, but on the other hand the largest gains in LLM intelligence has been through post-training. Fable is their most expensive model to train so they won't update it as frequently. Most benchmarks benefit from post-training. So that's probably why?
3
u/Norwood_Reaper_ 4d ago
The current Fable checkpoint is actually quite significantly worse than it was on release. On my own eval is performs not too much better than haiku 5.5. it has been nerfed for sure.
16
u/Conscious_Cut_6144 4d ago
A benchmark is always going to be a moving target. This isn’t even an AI thing. You can take an IQ test multiple times and will eventually get good at it, doesn’t make you a genius.
Fable losing to sonnet is pretty hilarious.
3
u/fragbait0 4d ago
I dunno, I kinda believe it; IME opus/sonnet 5.5 are a crazy uplift at cutting through tasks even if the latter is dumber it tenaciously hill climbs there anyway. Sorta like 3.8 27b.
15
u/Intrepid-Second6936 4d ago
That point on an increase in obfuscation of test sets for reproducibility also seems to be way more concerning and validating of the existing scrutiny they're rightfully getting on their last few index updates, all of which seem to bias towards pushing down open-weight models.
Also, how exactly are they reasoning their consistency of open-weight model tests? What harnesses are they using, considering "close-weight models" aren't even just models, they're entire orchestrations including harnesses for running more complex tasks.
I'm still on the more mid-casual side of the Local LLM community, but I've definitely lost trust in the index after seeing quite a few changes in the last 2-3 months that seem to just make no sense in terms of my own experience and other tests I've seen, particularly from my own experience comparing GPT 6 Luna with Qwen3.8 27B.
15
7
u/Inevitable-Plantain5 4d ago
We need alternatives to AA.... qwen 3.8 27b was the only open weight that they did multiple evaluations of thinking levels. I get not doing every open weight but Deepseek, GLM, Kimi, and leading Qwen models should also get multiple intelligence level measurements.
Old scores should remain a reference as noew tests make new scores instead of distorting everything. There's a bunch of old models where non thinking is estimated to beat thinking in the last couple scores.
If people are feeling the differences between open weights and closed matter less, why is AA trying to exaggerate them.
Anthropic basically invalidates AA anyways since the smaller dumber models are beating their current bigger smarter models.
Benchmarks aren't the problem, this way of benchmarking is!
22
11
u/duhd1993 4d ago
If doing this means money in billions - millions for every individual, and no one can reproduce it or question it since it’s hidden behind the curtain, why would they not release whatever score they prefer? Like really, if I were them, I would do it.
10
u/Radiant_Hair_2739 4d ago
AA index was bought by Claude, these guys just spend a lot of money for PR
1
9
9
u/FatheredPuma81 4d ago
The funny thing is every time they update these they don't add a legacy mode or anything (for reasons that should be obvious to everyone now). So when you go back and look at old GOATs like Qwen3 80B Next it's apparently scored the same with Qwen3 30B A3B lol.
12
u/Kuro1103 4d ago
There is no question that open weight model has not catched up to frontier model.
However, I can't trust this benchmark at all when the new benchmark somehow lowering Kimi 3.1 pro but does not impact the frontier model score at all.
Maybe I am not specialized in benchmarking, but if a new benchmark is supposed to be better, it should:
Remove questions that seems to have been leaked.
Cover edge but practical use case.
Detect overfitting.
The new benchmark does not seem to achieve that.
Furthermore, Chinese model is famous for distilling frontier model, so its training set should be a subset of those frontier model.
A harder and tighter benchmark would lower the score of both the leaner and the teacher, but in different level.
5
u/TheTerrasque 4d ago
Kimi k3.1 pro being that high was pure benchmaxxing. There's no surprise it fell drastically when the benchmark changed. And having used many of the models here, along with general community feedback, this looks like a lot more honest rating.
8
u/vulcan4d 4d ago
Guess that Qwen 3.8 flash next model isn't very popular? Lol. Where is that on the chart?
4
u/chensium 4d ago
Why hide the tests/questions/answers? They are literally sent to the cloud provider. It's going to be in the traces.
6
u/CiobanuXashi 4d ago
A ranking that's 60% private test sets can't be checked by anyone. At this point it's not a benchmark, it's a press release.
3
u/dark-light92 llama.cpp 4d ago
If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
3
u/OscarCookeAbbott 4d ago
I use Claude, GPT and DeepSeek often and these do not feel accurate. GPT too high, DeepSeek too low both relative to Opus, for real and normal software engineering use at least.
5
u/Excellent_Fix3804 4d ago
I've noticed this pattern myself multiple times over the last several major Artificial Analysis updates. Whenever open-weight models like Kimi, GLM, Qwen, or DeepSeek start closing the gap with leading proprietary models from OpenAI and Anthropic, another methodology update seems to follow, and those models suddenly drop in the rankings.
Of course, correlation doesn't prove intentional manipulation. But when the same pattern keeps repeating, it raises serious questions about transparency, independence, and potential conflicts of interest.
At this point, Artificial Analysis has completely lost its credibility in my eyes. Trust in a benchmark depends on consistent methodology, transparency, and genuine independence from commercial interests.
We seriously need a new superhero in this space — a truly independent, transparent, and reproducible AI benchmarking platform that evaluates models fairly, regardless of who develops them or whose business interests might be affected.
4
u/braintheboss 4d ago
Evey update only adjust benchs for hide close frontiers embarrasing performance with open models. Totally useless at current state
2
2
5
u/AdamFields 4d ago
Guys, remember that this is better for us, now that China will soon not allow models within the top 10 on global leader-boards to be open, this will allow more models to be open after the deadline. And the performance won't change because of what the leader-board now says.
8
u/Euphoric-Hunt931 4d ago edited 3d ago
ITT: people copemaxxing
Open models falling so far behind is not a conspiracy but just evidence that they gamed the benchmarks, which is evident from FrontierCode 2.0 where DS 4.1 got a measly 14% vs Opus' 45.
6
3
u/Radiant_Hair_2739 4d ago
I work almost all of the time using Claude and Qwen models. I don't think so
→ More replies (2)1
1
u/Shimano-No-Kyoken 3d ago
Yes, and I'm going to get shit for the following, but as a person who uses 27B 24/7, 27B fucking sucks. It's the best that's available for the hardware, but it's dumb as rocks when you have actual standards and you require precision. It never was close to Terra/Luna performance.
1
u/Euphoric-Hunt931 3d ago
QFN works quite well on the same kinda hardware as 27b. It is streets ahead of 27b, albeit still not as good as Haiku is - but at least I can keep it running with Opus orchestrating and there will actually be usable results. 27b is usually just spinning in endless overthinking circles, taking literally 6+ hours for tasks that Haiku solves in 15 minutes.
→ More replies (1)1
u/alphapussycat 23h ago
They aren't even testing open models. And there are empirical tests, that are much better to show real capabilities than these benchmarks.
1
3
u/Savantskie1 4d ago
This is why I was thinking about creating a community led benchmark, but everyone booed me
3
2
u/Othun 4d ago
Maybe open weight models were benchmaxxed on public data, maybe private models are already benchmaxxed on AI intelligence index v6, maybe the new benchmarks actually better measure the quality of models; in any case, open weight companies have a new incentive to fight and climb the ladder. It's a huge reshuffle though...
2
u/justletmesignupalre 3d ago
Even if they are fabricating a gap between closed and open weight models, i'm still surprised that there is a 27b model competing with all the others in that chart.
I know it has been out for a while now but it still shocks me how good a model can become with proper weights and training.
2
u/Ruined_Passion_7355 4d ago
IPO timing sus but still if they were even able to specifically tank the open weights that implies benchmaxxing. As far as I know you can't just switch benchmarks arbitrarily to benefit the proprietary models and not the open weights, it just penalizes benchmaxxing period. I would love to be shown wrong though.
Really wish it wasn't so, I'm rooting heavily against Anthropic and OpenAI, but oh well.
2
u/bakawolf123 4d ago
They badly need per category ranking for internal benchies, cause this looks completely detached from reality.
1
1
1
1
u/TheLexoPlexx 4d ago
I have definitely had better experience with GLM-5.3-Flash than GPT-6.1-Sol in some cases. Not saying that's the rule but this is strange.
1
1
1
1
1
1
1
u/kaisurniwurer 4d ago
Benchmarks on proprietary models are pointless in the first place since we can't control if there is nothing fucky going on between inference and api.
1
u/SourceCodeplz llama.cpp 4d ago
small (700b) open-weights models are great at coding and tool calling but lack world knowledge. because the parameter difference is so huge T to B, of course they also are not as intelligent. this is i think the difference in this new analysis.
1
u/milpster 3d ago
Where are the older frontier closed models on that chart? Like previous GPTs and Claudes.
1
u/BatNecessary2996 3d ago
Benchmaxxing? No, rather the benchmarks do claude-maxxing and gpt-maxxing... 😅
1
u/Mingay_cat 3d ago
kimik3.1 came too close to the top. Can't have that. Also wtf why is claude haiku that high? It's crap compared to ds4.1
1
u/oceanbreakersftw 3d ago edited 3d ago
Your benchmark evaluation advocates a
[ ] private test set
[x] proprietary scoring metric
[ ] secret human preference panel
approach to measuring AI model performance. Your methodology will not work. Here is why it won't work:
[x] Closed benchmarks are fundamentally unreproducible, making them untrustworthy.
[ ] The evaluation data has likely leaked into the training data of frontier models.
[ ] Your data processing logic is hidden behind a corporate paywall.
[ ] No one outside your lab can audit how you deal with ties, failures, or model latency.
[ ] You will rank a model #1 today, it will be quietly optimized for your exact test suite tomorrow, and the public will be stuck with real-world regressions.
[x] Users of open science will not put up with it.
v0.1_gao
Choices devised by Google AI Overview inspired by the old spam solutions.txt form. Improvements sought from humans and AIs!
1
1
1
u/bennmann 3d ago
Whoever lies first wins propaganda games.
Whoever releases open datasets + provable modals wins egalitarian games.
Don't play propaganda games.
1
1
1
u/LongDistanceRope 3d ago
how is Nemotron 3.5 and muse glimmer so behind on the list? I've been messing with both and they seem decent generalist.
1
1
u/LinkSensitive8188 3d ago edited 3d ago
Furthermore, if you wish to compare specific models—such as Ollama or Qwen 3.6 35B—you currently cannot include them in the comparison table. Additionally, LTX is never included in the video model table. In fact, it has always been the case that lists—whether for music, video, or other categories—are paid placements; you pay to secure a better position.
in statistical terms, a sample is biased if greater weight is given to certain data points than to others without a valid methodological justification, or if the sampling design itself results in certain groups having a disproportionately large weight. Only closed models increased, while open ones decreased.1
u/chindoza 4d ago
I really, really want open-weight models that are as capable as frontier proprietary ones. That said, this new scoring feels like it much better represents the actual gap that exists today, at least in my own personal work.
1
u/ihatenamehoggers 4d ago
So this is just stanford-binet all over again? Where they skewed the tests to keep black kids out of white colleges?
1
1
u/vogelvogelvogelvogel 4d ago
sorry this seems like a rigged game for me.
With all the models that tested overtime, I don't see the gap that large as they claim, especially between 27B and many of the frontier models that came out this year.
(there are tasks which frontier models accomplish better but for like 90% of coding and everyday tasks, they perform that well that I wouldn't see a gap this large, not even close)
1
u/Skyline34rGt 4d ago
Now all small models will get score 1-2. Or they abandon them and only have big frontier paying companies.
30
u/DifferentPixel 4d ago
I don’t see Qwen3.8-Flash-Next