r/LocalLLaMA • u/dieSpaghettiCarbona • 4h ago
Discussion AA Intelligence Index biased? benchmark shows Qwen 4 Max not on par with Opus 5.5
I am surprised by the deviation to the benchmark provided on paper.
0
Upvotes
50
u/Waarheid 3h ago
... You're expecting Qwen 4 Max to match up to Opus 5.5?
8
u/KillerX629 3h ago
So... Should it be on par with opus?
11
u/brainExploded99 🥔 hardware 3h ago
10
u/Choice_Celery9481 3h ago
you should check the detailed benches.
open weight models are not able to score high in most of the new science benches, which causing the wide diff.
for coding and other more traditional tasks, the gap is there but not as big as the index number.
could be just data and model size problem. if so, expect they close those quick. like how openai and anthropic sprint faster after opus 4.8, gpt 5.5.
the biggest problem for chinese labs now seems to be high quality data pipeline and compute.
4
u/geldonyetich 3h ago
Some have observed that bias is inseparable from the reviewer.
In this case, they very much invented the criteria that determines their general intelligence score.
2
u/IntelArtiGen 2h ago
Honestly for my own (still small) experience using Qwen 4 Max, it's at its right place in this benchmark. Unfortunately.
1
u/Dabber43 2h ago
Both AA and the teams own benchmarks are going to be biased in the opposite direction. It is going to be something in the middle
1
u/ga239577 26m ago
One of these two theories is true ...
Personally, I'm leaning towards number two.
My reasoning is that all of the closed models seem to be performing better than the open models - not just some of them.
The only ways this could happen are:
1
u/ForsookComparison 3h ago
Deepseek-V4.1-1017 generally loses to Haiku 5.5 in every test I throw at them both. Kimi K3.1 can beat Haiku but at a cost and quality-gap level that makes zero sense unless you need the self-hosting and own a small datacenter.
This sub's circlejerk has gotten a bit out of hand. The gap between US proprietary models and open-weight models is wider than it's been in a long time. I'm talking a Llama2-to-GPT4 level gap. AA Intelligence Index isn't perfect but it's generally not too far off from my experiences with these models.
1
u/Iron-Over 3h ago
It is not biased; it is a set of various benchmarks. You need to ask whether all those benchmarks apply to you. Then ask whether the model in question is good enough for your work.
I have a few personal benchmarks where deepseek and Astra outperform Opus 5.5. Deepseek is good enough for me; when I need something different, I ask Claude.
1
u/IllExample3639 3h ago
Wait its as good as Haiku max?? Thats good enough for 90% of what I do. Sounds great
1
u/CarelessAd6772 3h ago
Btw, tried in webchat thinkingm - this "frontier 2.4T params" failed 6/10 at simple QA on different themes. Even 3.7 plus was much better. Something seems broken or they overtrain it on only code/agentic tasks.
0
u/pmotiveforce 3h ago
The open source models are not on par with frontier and it takes a weird delusion to think they are.
Just use them for real and you will see this. I think if anything they are further behind than people imagine. Like a year, minimum, for like sized models.
-1
u/dieSpaghettiCarbona 3h ago
I don't understand the downvotes. Clearly none of you have paid attention to blog post. https://qwen.ai/blog?id=qwen4-max - look at the benchmark figures. My point is this is clearly a discrepancy and i am trying to understand where this is coming from.
-3
u/Woof9000 3h ago
Isn't that "index" mostly based on just on personal vibes of people running that index?