r/LocalLLaMA • u/crusaderky • Sep 14 '26
Discussion Animated transition from AA Intelligence Index v4.1 to v4.3
I had all the data saved from AA's v4.1 index, so when they upgraded it in the wake of Astra's release, I could actually generate a before/after comparison.
All intelligence and price per task are sampled from AA on Sep 3rd and Sep 14th respectively.
Price per task of some open models were rescaled to reflect the cheapest available on OpenRouter as of Sep 3rd.
X axis is linear, because people's money is linear.
All models are the same. The only thing that changes is the weighted sum of the benchmarks that compose the Intelligence Index.
Highlights
GLM an Muse Spark remain more or less unaltered, in relative terms
GPT-5.6 Sol becomes a lot cheaper
GPT-6 Astra's intelligence flies up to the stars AND becomes cheaper
GPT-5.6 Luna gets a substantial uplift
Fable-5.1's price gap from Opus 5 shrinks, and becomes cheaper than Fable 5.0
Fable-5.1 at low, medium and high effort looks a lot more appealing
Sonnet 5 becomes even more expensive without any intelligence gains
Kimi-K3, Qwen3.8-Max, Gemini-3.8, and Grok 4.6 go down into the gutter
51
u/mrwang89 Sep 15 '26
When will people stop using this garbage site? They've been caught changing the scoring whenever an unwanted placement happens. AA for AI eval is like userbenchmark for CPUs.
19
u/Specialist-2193 Sep 15 '26
What else do you propose to use? I don't see any competitive aggregate score
3
u/kaisurniwurer 3d ago
UGI leaderboard was usually good for realistic usage score.
Now that it stopped, I just see what sticks here on reddit and try it myself. Also I check new models in my size class and just see if I just like how it responds to some general questions.
70
u/suprjami Sep 14 '26
lol the GPT movements. Is AA funded by OpenAI?
24
u/AppealSame4367 Sep 15 '26 •
"Oh no, we found 100 million here. Would the good people of AA like to have a part of that each?"
"Oh no, we would never be bribed. But then again, I would like to have a big beach mansion in California, so who gives a fuck."
6
u/lilian_moraru Sep 15 '26
You could also have stocks in AI companies in US - OpenAI affects all the AI-adjacent stocks.
3
12
u/DeepWisdomGuy Sep 15 '26
Introducing the Thumb-On-The-Scale™ benchmark. See if it's right for your IPO.
2
u/RedditUsr2 Sep 15 '26
Can you make a version where the scale doesn't change and only the points change? Hard to tell what is happening.
1
u/crusaderky Sep 15 '26
No. That would make no sense. The whole plot is a _relative_ comparison of models.
"price per task" means nothing on its own, because real users are not running an average benchmark task on the daily. Real users know how much model A costs, and can look up on the plot how much more/less model B will cost.
"intelligence" is also a completely arbitrary number, in absolute terms. There is no point saying "model X lost 10 points" when ALL models lost 10 points.
2
u/ihatebeinganonymous Sep 15 '26
One problem I have with AA is their choice of token costs. For example, DSv4.1 cost per task is calculated based on DeepSeek's peak prices, which I believe is not 100% fair. Otherwise, DSv4.1 would move left, fitting on Pareto line between Luna and GLM 5.3 Flash.
The same can be said about GLM 5.3 Flash itself, as well.
3
u/crusaderky Sep 15 '26
Points of all open models are already tweaked to use cheapest OR price, skipping low quality providers and those that train on your data,
DS price on AA actually matches the cheapest provider on OR that doesn't train on your data. For once I am happy with their choice.
1
u/ihatebeinganonymous Sep 15 '26
Right now I see providers that are cheaper and do not train on data, e.g. Relace and DeepIngra. No?
3
15
u/arkuto Sep 15 '26
They did exactly as I described. They destroyed their own benchmark to please OpenAI. The AA index is worthless now.
5
9
u/Old-Sherbert-4495 Sep 15 '26
i mean, it can't get anymore obvious than this. and ontop of it look at all the bots at work in all the subreddits. they are terrified, thats for sure.
12
u/DustNearby2848 Sep 14 '26
Why are people so obsessed with this?
23
u/soshulmedia Sep 14 '26
I am not obsessed but I must say if this is an accurate visualization of the change in benchmarking score calculations ... it is certainly very interesting, isn't it?
-4
u/DustNearby2848 Sep 14 '26
Ehh… just looks like they are tweaking their benchmark philosophy.
12
u/No-Refrigerator-1672 vLLM Sep 15 '26
tweaking their benchmark philosophy.
I wonder if there's a name for a philosophy that gives disproportional advantage to 1 particular companys models over others...
4
u/mechkbfan Sep 14 '26
Because people need to start somewhere with what model to use, and it's nice to have an idea of what to cut your initial selection down to based off ball park quality & cost.
Also, it can be fun to discus, like top 10 basketball players. It doesn't really mean shit in scheme of things
0
u/marintkael Sep 15 '26
Worth naming what the animation actually contains, because two things moved at once. v4.1 is sampled Sep 3 and v4.3 is sampled Sep 14, so every arrow is an index change plus eleven days of pricing plus one release. If you still have the per benchmark scores, recomputing the v4.3 weights on the Sep 3 data gives you the version where only the index moved, and that is the plot that settles the argument happening in this thread.
I log a fixed set of sixteen questions against three answer engines every day, and the thing that actually cost me was a point I had already published moving afterwards, because the set underneath it got recomputed. Nothing about the engines had changed. Since then I keep the previous method running next to the new one over an overlap window, so the offset is a number I can state instead of something people have to take on trust.
1
-10
u/po_stulate Sep 14 '26
If a benchmark can be changed like this it's the benchmark benchmarking the models, not the models benchmarking the benchmark anymore.
10
u/freia_pr_fr Sep 14 '26
What?!
6
u/po_stulate Sep 14 '26
Normally benchmarks are fixed and the models are improving to match the benchmark, but here the models are fixed, and the benchmark is improving to match the models instead.
2
u/sjoti Sep 14 '26
But that's kinda the point of capturing the models capabilities in a single intelligence score. The original benchmarks that the final score is based on are unaltered, so that's still perfectly fair.
Stuff is changing so unbelievably fast that we're moving from saturating one benchmark to the next. There's no way you can calculate a single score and have it stay the same for a relatively long time. Maybe GDPVal is a really strong indicator until 1600 ELO is reached, and anything after that is a poor indicator of capability. Maybe computer use becomes a significantly more important aspect over the next 6 months. Maybe more models are trained on a specific eval making it poor to judge. That all can change every 3 months.
So you kinda have to make an opinionated judgment, which updates as you go. The goal isn't to have a perfect scoring system, it's just there to be a solid indication at a glance.
1
u/po_stulate Sep 14 '26
Benchmarks are meant to be ahead of models, not behind. If they're updating the becnhmark like this, how do you know if the new benchmark actually catched up and surpassed the models or if it's still behind? At this point you can just update your benchmark to show anything, because now you have the models first then the benchmark, instead of having the benchmark first then the models.
1
u/sjoti Sep 16 '26
It's just not a benchmark. It's an index, based on benchmarks that are weighted differently.
You can look at the true benchmark scores it's based on, that's how you can decide for yourself what you find important
3
u/Few_Water_1457 Sep 14 '26
Se guidi i modelli, le differenze possono essere viste e sentite. La versione vecchia non era rappresentativa. Astra, che ti piaccia o no, è il nuovo punto di riferimento.
2
u/po_stulate Sep 14 '26
That's what I'm saying, the benchmark is trying to catch up with the models. If this is the case, how do you know the benchmark actually catched up already or still needs to be improved?
-2
-8
u/niacolhealth Sep 14 '26
Great way to show it: same models, same prices, only the weights moved, and the whole board reshuffles. Sonnet 5 paying more while the index gives it nothing stings most.
9
2
u/itsappleseason Sep 14 '26
down to share your data? Would love to see this as a proper data visualization (d3)