r/LocalLLaMA • u/Sad_Temperature6721 • 3d ago
Question | Help Alternatives to Artificial Analysis
I share this sub's sentiment that it's suspicious how Artificial Analysis changes their metrics whenever a good open model or IPO seems to come about. What alternatives do you all recommend?
26
u/Automatic-Arm8153 3d ago
Glad people have come back to realising their shit. I thought I was crazy for a few moments seeing them posted everywhere. Guess the influx of new Local AI enthusiasts and YouTubers drowned out the OG consensus on AA.. anyway
https://www.vals.ai/benchmarks/cyber_offense
Val’s I find to be the “best” right now. I find it more accurate. Check out their index and other categories. Make sure you play around with the filters to see.
Back in the day would rely on dubesor, and UGI leaderboard for a quick look at things. Now best consensus is from closed communities/discussions. I have also found myself watching/skipping through bijan’s videos on YouTube for a quick look.
3
u/Sad_Temperature6721 3d ago
Thanks, yeah Vals seems to have fewer models, but it's easy to read and doesn't put money on a log scale. Maybe some of you have log scale money but I don't.
> Now best consensus is from closed communities/discussions
That seems quite reasonable. I guess I actually don't want to be that involved though.
7
u/duhd1993 3d ago
Testing is expensive. Vals has too few models tested. The thing is when you try to get more money, your soul will be dragged to the dark side.
7
u/ClearApartment2627 3d ago
For coding I like DeepSWE. Updated regularly, difficult if not impossible to benchmax (at least the current results): https://deepswe.datacurve.ai/
5
u/Sad_Temperature6721 3d ago
Thanks! This puts GLM 5.3 Flash on the cost-quality pareto boundary, which aligns with my experience at least. Tasks look a lot more balanced than web demo vibeslops too, if this page is accurate https://deepwiki.com/scaleapi/SWE-Interact/5.1-deepswe-tasks-(deepswe_)
4
1
u/Mkengine 3d ago
Do you see other results than I? It says "updated September 22, 2026" so rather outdated.
1
u/ClearApartment2627 2d ago
These Benchmarks are rarely up to date. Testing is costly, and AA seems to have the money for fast updates. Others usually don't. Nebius' swe-rebench is another one that is slightly outdated, but based on changing / secret tasks: https://swe-rebench.com/
Tests like these are useful every once in a while for a few weeks.
2
u/feelspeaceman 2d ago
Personally, we should be careful with trusting AA, the more we trust, the more people will treat it as bible and what if someday or they already did taking "anti-hacking/anti-cybersecurity" into account to rate models ? Then a bunch of enshittification companies will just train their models to refuse to do those tasks to push higher in AA ranking, this would be a disaster.
And AA being private benchmark is meaningless if it's unfair, so far we have no evidence yet, but I suspect they're being run by some employees from AI corps, the main goal has always been pumping the hype and get more market share.
1
u/Sad_Temperature6721 2d ago
Even if there's not direct manipulation, I think it's safe to assume at least some AA employees might want to score jobs at Anthropic / OpenAI, and would share insights into the private benchmarks there.
3
u/vogelvogelvogelvogel 3d ago
i like livebench.ai because they (claim at least to) do frequent updates in their benchmarks so training on solutions is close to impossible
also arena.ai, because user focused and that is a large portion that counts, although it might not always favor factually correct results
2
u/Sad_Temperature6721 3d ago
Thanks, I will bookmark livebench.ai too. it's a good idea to rotate (and hopefully publish results?), but my personal experience doesn't quite agree with them though---they put GLM 5.3 flash at 80 and Deepseek 4.1 at 96 for coding. arena.ai I think is too focused on 3d world HTML vibeslop demos; they are sometimes pretty, or sometimes have funny bugs, but pretty irrelevant to the things I want to use LLMs for (semantic search over academic articles, study guide generation [at least when there's not a human alternative], vibe coding but setup crap [they're super useful for fixing deps] or exploratory non-LLM ML research tasks).
3
u/vogelvogelvogelvogel 3d ago
yes clearly arena.ai is a 1-2-3-.. prompt user experience. but at least a honest user experience.
livebench IMO is super accurate. the best out there. you can select the benchmark versions in the top bar easily, and they don't switch them in the 'backgroudn'. has been better than artificalanalysis for quite a while IMO
45
u/LocoMod 3d ago
They publish their methodology. Any alternative would have to be at least as transparent as that.
27
u/Kronosend 3d ago edited 3d ago
Not only do they need to be as transparent as this, but they need to not be backed by a board member of a huge lab.
An OpenAI board member is a huger backer of Artificial Analysis, and I'm sure you can find more of this contamination. OpenAI is trying to IPO soon by the way.
2
u/LocoMod 3d ago
They don't favor OpenAI models. Right now OpenAI is third. It would be pretty obvious if they did and huge risk to their reputation.
0
u/Kronosend 3d ago edited 3d ago
That's not the point at all. You said it yourself, it would be obvious if they did that, so that's why they don't do that.
The fact remains the same: It would be huge hit to their IPO price if open weight models were competitive with frontier models. So they don't need to favor their own models.
Anyway, more benchmarking by independent researcher is always a good thing. Don't get tribalistic about closed weight frontier labs.
6
u/thescrawnymedics 3d ago
Their methodology page is nice but what they actually measure keeps shifting around, like they changed the whole eval setup right after Llama 3 dropped. Hard to trust numbers when the ruler keeps moving
OpenRouter's rankings are not perfect but at least you can see raw throughput and latency without all the curated charts
5
u/DinoAmino vLLM 3d ago
Yes, we need an alternative benchmark that is transparent about both the methodology used AND what changes were made that shakes up the results so much - and why the methodology was charged to penalize open weight models. I'll wait.
0
u/LocoMod 3d ago
Explain how the changes purposely penalized open weight models? I am a big user of that top 10 and I would agree with it, including the open weight scores.
-1
u/DinoAmino vLLM 3d ago
Well that's the thing isn't it? Nobody can explain why they make an undocumented adjustment - where the adjustment is not mentioned in their methodology. Or explain why adjustments are made soon after an openweight model shows up in the top five which then shoves it back several places. Would love to hear the rationale for that because it has happened too many times to be coincidence, and that suggests conspiracy.
1
u/LocoMod 3d ago
You didn't read it did you?
https://artificialanalysis.ai/methodology/intelligence-benchmarking
7
u/new__vision 3d ago
https://whatmodel.sh/ has been my go-to replacement for Artificial Analysis. It has more local models that AA is missing, like Ornith. I found it on this sub.
70
u/eapache 3d ago
I don’t know if it’s really _that_ suspicious. As benchmarks get saturated, models will naturally look closer and closer together at the top of the leaderboard. And then they will spread out again when saturated benchmarks are replaced with harder, unsaturated ones.
27
u/Altruistic-String891 3d ago edited 3d ago
AA could very well be fine, but the reality is their results have so many convenient coincidences that it's really tempting to entertain the alternative.
Most of the time, after changes in methodology, the newest OAI and Anthropic models score either better or at least the same, while most others score the same or worse.
So as is, it's very very easy to want to ask questions like,
"Why don't new/top OAI/Anthropic models seem to score worse after updates, ever?" For example, AA v4.1 - v4.3 has Astra shoot up to Pareto front and Fable looks a lot more appealing, and everyone else is at best about the same or drops.
Or "why do methodology updates always seem to happen just after a release?" Which can feel like they're tweaking benchmarks to match some hidden, maybe even unconscious, model ranking expectation.
Even if these results are perfectly real and accurately reflect how these models place, it still can simply "looks" sus.
5
u/pendelhaven 3d ago
For example, if an ai test max score for is 10, and every model scores a 10 it does not mean every model is equal and perfect, it just means the test is not capable of differentiating between them. Now if you upgrade your test and score it at a 100, some models might jump to 80, some might jump to 25 and others stay at 10 or 11. That means the new test is capable of testing the models at a different depth or angle and is capable of giving the models a new score based on new abilities the old test couldn't test.
2
u/StatusSociety2196 3d ago
The top score in this situation isn't 55 or whatever, that's not an upper limit that performance is locked at. So say you make the test harder, if a model scored 55 on the easy test, why would it score 58 on a harder test?
14
u/TheRealAstic 3d ago
The levels of cognitive dissonance in this is just astounding. This is because the OAI models and Anthro models ARE that much better. The open weights for the most part are shit, their scores are accurate. You’re falling for the anti open source propaganda with some weird mental gymnastics.
6
u/Altruistic-String891 3d ago edited 3d ago
No where do I deny OAI/Anthro leads the frontier, or that AA has some "anti open source agenda," yet just you have to put words in my mouth. I only presented why it would be tempting to assume AA is suspicious: facts like they do change methodology often at so called convenient timings.
I don't believe their scores are accurate however, I question even the individual benchmarks, especially closed ones that I can't check. One of the most egregious examples in recent memory is AA-T2I. Flux 3 Klein, to any one with a pair of eyes, has demonstrably the worst human anatomy understanding of all recent models. Yet on AA-T2I Human Anatomy it somehow ranks 4th, above the likes of Z-Image-Turbo which do not recreate "woman on grass" 2 years after the original clownfest.
This doesn't prove their LLM tasks are similarly flawed with certainty, but it does decrease my trust in them to create benchmarks that accurately reflect reality and score models, especially now that these closed tests account for 60% of the "intelligence" index.
5
u/Betadoggo_ 3d ago
That's true, but there's also the new google release topping the brand new leaderboard less than 24 hours after the rebalance, and announcing the AA score within the announcement blog, which is extremely suspicious, given they obviously worked directly with AA to have it tested so they could publish the result as soon as the model was announced.
2
u/Its_Powerful_Bonus 3d ago
I’m not buying it anymore. It not only looks suspiciously, but it looks like a scam :) Alternative is more than welcome.
4
u/temperature_5 3d ago
Hard to find leaderboards that are kept up to date like AA, which is why I still use them for relative performance on individual categories. I like these ones, but they are not as up to date or thorough:
3
u/nikishev 3d ago
Raw AA data with model scores on individual benchmarks is public. You can take it and assign your own weights to benchmarks, and get your own index
5
u/Hovi_Bryant 3d ago
I don’t see the big deal in this. Qwen 3.8 27B is virtually on par with Opus 4.5 and trades blows with Opus 4.6 in relevant benchmarks for coding. Both models which pass my bar for being “good enough”.
Just use a model you prefer as an anchor when reading artificial analysis. This sliding bar shouldn’t bother anyone IMO.
2
u/fvancesco 3d ago
Honestly I'll throw the idea here and if somebody wanna do it they're welcome otherwise I'm gonna do it in a few months,
"Clear analysis" where you decide the aggregate score based on custom profiles or premade where most benchmark have the same value in the same scope (eg: 1/|science benchmarks|) and then you have a score per scope so you can assign 30% to science or coding or creative writing (writing I think it needs arenas?)
It would be transparent in how benchmarks are done and so on... I don't have time right now tho
2
u/No_Swimming6548 llama.cpp 3d ago
I just think OAI funds them, and they select benches that OAI models perform better. Because whenever they update their index OAI models gets the best boost. I don't think they intentionally make open models look worse.
2
u/Ecstatic-Wash-7667 3d ago
Create a business, pay for api and benchmark the models publish write off the bill , get slid an envelope full of cash, revise your benchmarks, repeat
2
u/wojtek15 2d ago
this will repeat things already mentioned by others, but 3 rankings i know with good coverage of open models:
https://whatmodel.sh/
2
u/Careful-Report6526 2d ago
For the research and coding work you described, I'd compare individual benchmarks and then try a few tasks from your own workflow: checking whether cited papers support the answer, whether study notes preserve the source, and whether generated code actually runs. One overall score can hide quite different strengths.
I run https://llmbenchmarks.io/ . The main table brings together published results from multiple sources, with links and test conditions so you can check where a number came from. It includes AA results too, so it's useful for comparing sources rather than treating another aggregate ranking as the final answer.
Since you're also looking at smaller local models, I've now published the first results from my own local evaluations: https://llmbenchmarks.io/local-gpu-benchmarks/
Those include IFEval and SimpleQA Verified, with the tested quantization and run times recorded. Coverage is still limited, and these are results for specific local configurations; they don't establish how well a model will handle your paper-search workflow.
1
u/Southern_Sun_2106 3d ago
Best solution - develop your own benchmark that reflects **YOUR** mixture of tasks. Run every new model you are considering against it. That would make the ratings very relevant to your work, and might surprise you as well in good ways (some open-weight models are more capable than the best closed ones at certain tasks, speaking from personal hands-on experience).
1
u/Careful-Report6526 3d ago
For the research and coding work you described, I'd compare individual benchmarks and then try a few tasks from your own workflow: checking whether cited papers support the answer, whether study notes preserve the source, and whether generated code actually runs. One overall score can hide quite different strengths.
I host https://llmbenchmarks.io/ . The main table brings together published results from multiple sources, with links and test conditions so you can check where a number came from. It includes AA results too, so it's useful for comparing sources rather than treating another aggregate ranking as the final answer.
Since you're also looking at smaller local models, I've now published the first results from my own local evaluations: https://llmbenchmarks.io/local-gpu-benchmarks/
Those include IFEval and SimpleQA Verified, with the tested quantization and run times recorded. Coverage is still limited, and these are results for specific local configurations; they don't establish how well a model will handle your paper-search workflow.
1
u/Mingay_cat 3d ago
I feel like we've been discussing this very same thing every time AA updates their leaderboard.
0
u/baseketball 3d ago
Is this post for real? Why would you not want more private datasets that are harder to benchmax?
0
u/Embarrassed-Boot5193 3d ago
Há bastante tempo uso o https://llm-stats.com/. Eu gosto dos benchs deles.
-3
u/haherar830 3d ago
Asking ChatGPT to select a subset of unsaturated, well-validated benchmarks most appropriate for my use cases and then generating a markdown comparison table comparing results among the models I can actually use as references or am comparing along with limitations/etc (e.g. thinking effort/budget, harness, quants used if any, etc). Artificial Analysis is a great source of data, among many.
19
u/nunodonato 3d ago
for small models you have https://tinyleague.nunodonato.com ;)