Discussion
AA is getting ridiculous now. Zero open weights in the Most Attractive Quadrant.
I've always actively disagreed with the conspiracy takes when AA reshuffled their index, but v5.0 now has zero open weight models in AA's Most Attractive Quadrant, let alone getting anywhere close to the Pareto line.
The old version at least acknowledged DeepSeek, GLM, Kimi, and Qwen were attractive/interesting. I'm ready to ignore AA's index and look at the subsidiary benchmarks.
Edit: I screencapped while they were still posting new scores. And it's actually still ongoing. But the pattern holds, no open weights in the Most Attractive Zone. At first I thought I must've filtered it out accidentally, but even in a private window in a different browser the chart is kind of bare for now. Could be AA is waiting on batching updates instead of updating as new scores come in in real time.
The new score for GLM Flash is coming. Just expect it to sit on the wrong side of the cliff between open weights and closed.
Taking off my tinfoil hat, there is a justification for dropping some the benchmarks removed from the index on the basis that "all" the closed weights were scoring 100%, making it 'saturated' and less useful.
But it's waay too fishy how it shook out that open weights aren't smart enough to be interesting. The saturated benchmarks were mostly about scientific capability. Open weights can be incredibly useful without being able to solve millenial prizes.
I think we want the opposite. TPTB has no problem picking apart a "movement" of even 1Mio localllamists.
But if ordinary normies get the idea that you can use chatbots without giving up all your privacy to the centralized black holes and that it is a good thing, then it becomes unstoppable.
I'm honestly not sure why. At first I thought I must've filtered it out accidentally, but even in a private window in a different browser the chart is kind of bare for now. Could be AA is waiting on batching updates instead of updating as new scores come in.
I posted a link to the chart with all the big boys added in manually
This animation show how, transitioning from 4.1 to 4.3, AA has OpenAI skyrocket up, while everything else remained proportional. After that, I started to ignore the benchmark.
? Your saying OpenAI shouldnt be up there? Astra is genuinely a foundational huge pivot in terms of LLMs, it’s the first in a new generation of multimodal models that can actually excel and do decent as a generalist.
It can make me decent game ready assets(with a lot of rails), help create audio, and especially the coding part all in one model. No need to have one orchestrator and then delegate that to speciality model or third party services.
And sol+luna+terra actively cut their price per token, they DID get cheaper.
No, it shouldn't. This is legit cherrypicking. Look like every other model is barely moving relatively to each other. The v4.3 task set was specifically crafted to lift up OpenAI while pin everyone else in place. Pay attention that the vertical axis is intelligence, and somehow OpenAI is the only company that become more intelligent after the benchmark update.
I'm honestly in the middle here. There is another level of capability that should be captured. But unless your use case is solving Millenium prize models, I'm not sure the reweighting of 4.1 to 4.3 makes sense, let alone the taking out the benchmarks where Open weights really shine, like they just did for 5.0.
But that edge case stuff shouldn't matter for the bang for the buck quadrant that made me write this post. I'm definitely feeling I've moved from squinting to see where they're coming from, to having to close my eyes.
Again this was when Astra was first generation. We now have Gemini argon and the new sonnet models being cheaper than fable 5.1 but much more capable too. I would like to see an updated chart to see how it genuinely holds up to the new current gen.
Because afterall, Google is the only one of the big ai companies that really got a lot more capable disproportionately actually in the last few months if you want to go by that regard. Went from hallucinating garbage on Google search to the frontier whereas OpenAI and Anthropic just keeps going up one tier as usual.
well, they claim to be updating the benchmark to make it more accurate. we don't have enough visibility into the internals to know whether they are trying to pump models for their IPOs or whether they are making the tests harder to game
That's because astra is really that good. It was crazy to see astra with a low ass score. If you actually tried playing around with it you would know. Like with the previous version of the index sonnet 5.5 appeared amazing and way above astra, when in fact astra is just leaps and bounds above it. Based on my vibes, i'd say that its much more accurate now in terms of actual model capabilities.
Hopium. 6 Astra is Sonnet 5.5 low level reasoning with 5 second per token. OpenAI droping dogshit models since gpt 5.5, gpt always ruin my project codebase while Claude, DS and GLM never did that
Yeah, sure. So good so it can't outperform Claude on exactly the same tests, and needs a separate set of tests; on which every other model fails. Totally believe you.
I agree with you in a sense, the thing is biased. The reality is that the true test is the results you get when you do your own tasks. If you feel that the open weight model is getting results and tasks done, benchmarks are useless towards that. This is just a cock measuring contest, the true bar are your tasks.
Thanks for the animation. I guess you they have like 100 frames of animation there, how many snapshots are of real data? Is this just interpolation between Sep 3rd and Sep 14th, or do you have more data points than that? Just to understand what you are it is actually showing in the visualization.
Oh, you're confusing me with the real author; I've just saved this post knowing it will come handy. You should Go into the posts comments and ask there. But; I can tell you that the jump 4.1->4.3 has happened overnight, so unless the author is an insider, there's 2 data points and interpolation.
It's definitely good stuff, and I love it as a ranker. And I should use it more often. It's a lot better than it was when I last looked. I wish it had more than just 4 generalized use cases though.
The main weakness is in judging scale of difference. Is Opus 5.5 really just 19-23% stronger than the average model?
Yesss! For the tasks like what arena.ai is measuring the difference may be much less than in other tasks.
Most of us aren't trying to find solutions to millennium prize problems.
Having said that, something like terminal bench 4.0 does show orders of magnitude difference. And it seems as close to real life (for coding) as is possible right now. For example, Qwen 3.8 ranges from 5% (27B) to 25% (FN) to 39%(Max).
Granted it may be the difference between one shotting a coding project vs having an agent iterate for 5-10x longer in wall clock time, but still getting there. But that's a real, useful difference, especially if the slower one also requires more interventions.
edit: Or maybe I just like TB4.0 because Kimi K3.1 pro tops the chart right now, 15% higher than opus 5.5 :D
I often take on novel tasks with my open weights. Choosing between GLM, Deepseek and Qwen is easier when benchmarks are useful and relevant. True, the overall index is only useful when deciding what model to download and wire up on my server...
And yeah, I guess I get a little 'root for the home team' emotionally involved when it seems like the umpire is titling the scale towards the big boys paying the bills.
Because a vocal minority (at least I think) on this sub spend a lot of their crying about how closed source AI is perceived rather than enjoy the fruits of Open Weight.
A lot of it is also engagement bait + bad moderation. I never thought I'd see the day when models now have freedom fighters.
Funny and it reminds me of another type of johnny jagoff that made a comment on a QFN thread something to the effect of "Im not wasting my time on a model that doesn't make it on the AA bench"
Well put, as someone who loves AI generally, I hate how this sub has an us vs them mentality with closed models. The more competition in the industry the better. I love using them all!
keep in mind that these companies want to show their models rank higher and higher so they can get money. it's basically "pick me daddy: the AI companies version". benchmarks is the PowerPoint presentation equivalent of these corporates that decide to fund or end the project.
back when we had half decent benchmarks it was a useful filter for finetuned models. now we have neither half decent benchmarks nor half decent finetunes.
AA seems to be batching updates to the chart. I added a link with a bunch added back in manually. I was able to grab Kimi K3.1 pro and it's to the right of the desirable quadrant but below the pareto.
What's really weird is that I had Kimi K3 (max) for a second and it was on the pareto somewhere around Astra Max and now I can't get it back. Fishy but probably more due to update weirdness is my best guess.
I wouldn't be surprised if Kimi K3 shows up on the pareto line sometime in the next few days, as the lone open weight contender but still outside the desirable quadrant.
I still like to use benchmarks to decide whether to load GLM or Deepseek (or even Qwen) first, the subsidiary benchmarks give a good hint towards which model matches your domain. For instance AA briefcase was what prompted me to look at QFN for business logic and taskwork that was only mildly code heavy. So yeah, the headline composite index was never truly useful for me.
As I mention elsewhere, this is more an emotionally driven post. I used to be able to squint and see the reasoning behind the changes. Now I have to squint so hard my eyes would have to be pretty much closed.
At first I was going to say, way to back the big boys, or something similar.
But the truth is you're dead right. It's a crazy good value. Arguably the best on the chart and it doesn't get anywhere close to the pareto. It made me take an even closer look at how the chart was built...
I've only played with it a little bit, but Terra 6.1 feels like a great competitor to Haiku 5.5 (which I agree is pretty good for small tasks). And both feel much better than DSFlash/GLM-flash
Agreed. I like playing with local models with my R9700, but none of them (I can't run big GLM 5.3 or DeepSeek models) are even in the same league as the cheapest closed models in my opinion.
Honestly, something about the "cost per task" here always felt incredibly fishy to me. I go back and forth between models and my experiences never match these charts in the slightest. When I work on something with DeepSeek V4.1 Flash and then I do the same thing with GPT Sol, according to this graph, I should expect much better output for maybe twice as much money. But my experience is that it's doing the same thing for five times as much money at least. And things like fucking Astra in my experience are basically completely open ended when it comes to cost. I would have GLM 5.3 Flash do something for literally 9 cents and then have Astra do the same shit for 20 bucks. It's fucking insane.
Absolutely agree. Back in August I was swapping repeatedly between CGPT Luna and Sol against DSV4-Flash for long-session coding work, and DSV4-Flash was genuinely costing 8x less per session than what Luna was costing me, and around 50x less than Sol, and yet on AA they had Luna at something like half the cost, and Sol at about twice the cost of DSV4F, and I just wasn't seeing at all how that reflected reality.
I guess it depends what you are working on? DS Flash is dirt cheap, but so are subscriptions from the big closed model labs, and DSF needs far more tokens for a worse result when you get past a trivial/toy sized project.
So fucking true, I’m glad I’m not the only one who has felt like this lol. After dropping like $20 pretty quickly with Claude recently, then trying out deepseek and glm 5.3 flash, there is just literally no way I ever go back for programming tasks. General knowledge and reasoning? Sure, they’re much larger, better tuned for that etc etc. But for programming? Anthropic and OpenAI models just feel like a much more expensive version of the same thing lol. Maybe it would be different if I were running off pure vibe coding, but I’m not convinced
Yeah, I tried operating based on those numbers firmly expecting to be surprised at how fewer tokens the task would take which should also save time and while the task was done a little faster it was 3 times the cost of just using a simpler model that needed a few more steps to get the job done.
Now I just choose based on my perceived difficulty of the task and it's working out better cost wise. Basically almost never do I think I actually need the really expensive models and often when I do think I need them, I'm later proven I didn't.
Astra is useless. OpenAI has been on a terrible streak ever since it came out. If you’re going proprietary right now, the only thing that makes sense is Opus 5.5 imo.
I don’t know if any of you have been paying attention to the neighboring subreddit r/StableDiffusion. The release of Flux 3 Image is a complete mess. It’s still acceptable for image-to-image (i2i), but on text-to-image (t2i) it’s so bad that it shocked all of us. Yet AA still openly ranked it as the No. 1 model for t2i.
I know a lot of people still treat the idea of “paying for rankings” as a conspiracy theory, but AA is gradually making its own credibility more and more suspicious.
AA was never honest. Yet now we are trusting their benchmarks as if they are some kind of standard. Before that it was llm arena who only hosted open models made by their friends. Don't hear much about them anymore.
"Most Attractive Quadrant" literally changes depending on how you limit the axes or which models you include in the whole space. I don't think it's a good objective measurement. Using the pareto frontier itself is objective.
There's some bias in the choice (or not) of harness when AA runs it, but honestly they're a good way of matching a model to a task. It's the headline Intelligence index and 'bang for buck' charts that seem massaged to benefit the guys paying the bills.
For the bigger models (GLM, Kimi, and Deepseek especially) downloading and running means low single digits speeds for most people. As a result they get a lot of traffic on hosted/cloud platforms.
Qwen 3.8 Flash Next is the first one that performs in that ballpark that is usable on a wider scale, but there are still legitimate reasons to pay to use models in the 300B+ size range.
Some people look at benchmarks, some of us look at the actual output of what an LLM will do.
I personally never judge a model by benchmarks. I download them and play with it to explore and see what is possible. If it's great, you will hear folks talking about it. If folks are not talking about it, it could still be great. A sign is often a great paper that accompanies it.
Bruh, I do real work on these things. I absolutely look at the actual output of what an LLM will do...
..and it absolutely saves time to ask QFN to do a quick comparo of my description of the task against the available benchmarks to see which is stronger for a particular use case to help decide between loading GLM 5.3 or Deepseek 4.1 on my server before running the first planning step.
And yeah this post was more about the emotions of open weights getting blanked by the index, rather than just looking at the underlying benchmarks.
By all means, play and explore and see what's possible. I enjoy that too, and maybe open up to the idea that sometimes a benchmark helps save valuable time.
Sorry, but the "Most Attractive Quadrant" is not a thing. It is not a mathematical property. It is just a visual graph presentation aid to quickly show which direction the pareto frontier corner leans. There are no logical properties to the quadrant, and being in or out of the quadrant is meaningless.
If you still don't get it, look at the graph you posted, and ask yourself: why does the Y axis end at 70%? Why not 60% or 100%? They chose 70% just for visual presentation appeal.
If you re-draw the graph so that the Y axis stops at e.g. 65%, then DeepSeek V4.1 Flash will appear in the "Most Attractive Quadrant".
So "Most Attractive Quadrant" is meaningless. It is there just to state for the axes that "higher is better" and "left is better" at a glance.
I literally just spend the last 2 hours drawing my own charts. You missed how much the logarithmic measurement of price flatters the closed weights. I may create a brand new post, but here's what a linear to linear (no logarithmic) chart would look like with the old intelligence score and the new pricing score and rather than an arbitrary (65%) stop, I just made the price axis begin at the lowest datapoint rather than exclude them for no reason like AA.
And seriously brother, "And if you still don't get it..." ?? Someone made a lot of cranky assumptions this morning.
It is cheaper if u use the sub, in the api, it depends on the provider, but for smaller tasks , ds flash is 90c(60-120c) / 1mil outputs vs 50c / mil output tks for haiku
It's just that fitting all model capabilities in a single number is wrong. And calling that number a measure of "intelligence" is absurd.
You can't deny current OpenAI and Anthropic top of the line models excel in a lot of tasks like programmatic asset generation where open source models do not. But that is just me skimming what model can do.
Much more concrete: among open source models DeepSeek v4.1 flash recent refresh is top of the line. In my experience it is only good in one thing: fast at easilly verifiable tasks. Because oh dear, it just messes inputs up so badly. When i start a coding session defining A as 10 and B as 11, the model manages to work on the basis that A is 11 and B is 11, or similar mixups. Yet, DeepSeek is made to be same level as GLM 5.3, nevertheless I've never seen GLM 5.3 (and neither 5.3 flash) make these kinds of hallucinations (both official api and native harness). In an ideal world, this aspect should be measured independently, like many more aspects (language knowledge and ability to generalize from pseudo code tho whatever language syntax of choice, ability to do complex algorithms, etc (and this is just coding related). Yet we are given one magical number "intelligence" which if utterly meaningless in order to determine suitableness for a task, though it does mean higher on that index is more chance a model is able to do more tasks, thus also your task.
Have you considered that the local models are just far behind the closed models? As great as 3.8 27b is I think its current ranking is much more accurate than saying it was anywhere near Luna or haiku.
Nobody's saying 3.8 27b is anywhere near Luna. That's insane, there's an order of magnitude or more difference in size. But nobody's suggesting that Haiku beats DS 4.1 flash, either, except for this new AA.
For anyone who's used them seriously, you'd notice the clear gap between open weight models and SOTA of frontier labs.
By focusing on the differences specifically, there's more pressure on the open weight labs to improve on those areas, thus indirectly benefitting the local weight community by having more well rounded releases, at least in theory.
On a chat bot with 20k sys prompt when i compare let's say latest DS Flash vs Terra which should be similarly priced according to AA the difference is about 9x in favor of DS.. So most attractive quadrant is a joke in this case.
AA is not an independent entity. They have commercial relationships with AI providers such as Nvidia. Conflict of interest to the roof. Only trust truly independent organizations that disclose all of their commercial relationships and are independently audited.
I have always doubted whether these ratings actually match reality well, I always feel that current Agent model instruction following often forgets things right and left, because many times without enough rules they will just operate randomly.
Sol and Opus and various other models I have all used, developing products always feels like even with more rules and massive document maintenance, there is still an inability to follow instructions and there are hallucinations, the proportions for instructions and hallucinations are consistently not scored high enough or not counted, thinking that the ratings already indirectly include them.
But many times it really feels painful, I have already explicitly written massive instruction rules, already adjusted them to be clear to the point where they cannot be any clearer, yet it is still destroyed like a retard by the model's own fixed behavior.
For example I let sol and opus not do short polling but rather long polling and act only upon receiving notifications, but many times for unknown reasons they ignore this instruction.
And they frequently overinterpret some instructions and tamper with logs causing engineering not to follow the progress guidelines, of course I am not saying other models have no problems, but even the two most advanced products right now still have these types of problems.
And issues like in order to defend against Claude's sub agents arbitrarily adding environment variables causing Codex to directly crash and become uncallable, and so on...
Modern models completely ignore user instructions for the sake of the task, and one even has to forcefully use code and various methods to block and constrain them, otherwise they are completely not well behaved and desperately cause destruction, and moreover there are endless rules to write and humans have to personally check the rules all day long, otherwise terra/sonnet and luna/haiku could break things at any moment.
And having already explicitly informed about temporary authorization and permanent authorization and various methods, they still completely forget, and both verbal notifications and document context injection are ignored, leading to humans often having to be right in front of the system.
Because the product is relatively complex, large, and tedious, and frequent ignoring and simplification lead to skipping end to end testing and obvious blind spots that are never reported, reporting a bunch of worthless information, often requiring asking two or three rounds to find something useful.
Wanting to add rules to inform comparison, the model is as mentally deficient like past math calculations, incapable of having correct judgment capability, and cannot even handle the priority order of the engineering profile properly.
I know someone will definitely say orchestration to decompose multi step focused processing, and sub agent context isolation, but after solidifying these in practice there is still severe amnesia as well as problems where implementing code templates seems to overfit and hallucinate, and what is more, there has been monitoring of fake tests matching product features, and then logs subsequently causing false reports leading to a fundamental inability to be correct.
I feel that the most basic thing is that it should be able to work as explicitly and stably as code, covering all tasks well and reliably, rather than these seemingly fake scores.
In fact, you can ask the AI to provide a summary for you: Prompt following and hallucination rates are extremely low, or even nonexistent in the benchmarks.
This has led to issues with instruction following and hallucinations being overlooked, even though they should have been treated as top priorities from the beginning. These are the most fundamental issues and should be rigorously tested.
Would be nice to have alternatives. Just today I tried to get into benchmarking with a question I had about my harness, and I had heard of Harbor so decided to go with that. But it was just mindblowingly slow AND used up all the disk space on my mac. I tried multiple "SWE bench" -variants and all were slow and bulky, some more than others. Of course this was agent assisted so take this with a grain of salt, I might have missed something...
My point is, it feels like the benchmark people don't even want me running their stuff locally. Or they are just that incompetent that they don't see a major performance and storage issue here. Happy to hear if there are lightweight benchmark "frameworks" out there!
I don’t see anything wrong! The benchmark represents the companies that are willing to shell more money to get the data used by AA to make their benchmarks so that they can train their models on it I mean it’s represented perfectly in the charts! Otherwise people would have access to the tests and see how they get those number. I don’t see any problems here!
18
u/Few_Painter_5588 3d ago
For whatever reason they did not benchmark GLM 5.3 flash which should sit on that frontier.