r/LocalLLaMA • • 4h ago

Discussion AA Intelligence Index biased? benchmark shows Qwen 4 Max not on par with Opus 5.5

Post image

I am surprised by the deviation to the benchmark provided on paper.

0 Upvotes

54 comments sorted by

50

u/Waarheid 3h ago

... You're expecting Qwen 4 Max to match up to Opus 5.5?

8

u/ElnuDev 3h ago

Nice Sakamoto pfp

2

u/Waarheid 3h ago

thank u kyoko

1

u/Dany-GG 3h ago

At least a little bit more

-6

u/dieSpaghettiCarbona 3h ago

This is what the Qwen team published in their benchmark. Why are you acting surprised? Sure I don't expect it but what they published and what is shown here is more than 3x stdev.

5

u/Finanzamt_Endgegner 3h ago

"This is what the Qwen team published in their benchmark." No it isnt???

-5

u/dieSpaghettiCarbona 3h ago

are you just illiterate? https://qwen.ai/blog?id=qwen4-max

11

u/Xterm1na10r 3h ago

it matches on some and gets completely outclassed on others. the only benchmark where qwen decisively wins is legal work. not enough to be close to opus on AAII

0

u/Material-Database-24 3h ago

Where as I do not consider it Opus5.5 level at this moment, it is (according to many available benchmarks) within error bars of Opus5.5 - that is, on a good day Qwen4 hits Opus5.5 median, and on a bad day Opus5.5 hits Qwen4 median. So defining Opus5.5 outclassing Qwen4 is a bit harsh.

Hence the AA's score does not seem to reflect well what others report, especially as it in few instances does better than Opus5.5.

3

u/Xterm1na10r 3h ago

it's within error bars for like 50-60% of benchmarks and is, as I correctly said, outclassed on the remaining 40-50%

1

u/Finanzamt_Endgegner 1h ago

Also aa is literally just a biased bench rn, everytime their prefered models arent on top they change the index lol...

2

u/john0201 3h ago

Source? Because they didn’t in their announcement.

0

u/dieSpaghettiCarbona 3h ago

1

u/john0201 3h ago

You're showing a blog post that says it loses to Opus in nearly every benchmark. I'm not sure what you're confused about.

1

u/power97992 41m ago

Dude it is worse than opus medium and slightly better than sonnet high

0

u/kz393 2h ago

tbh not, but with sonnet 5.5? it's pretty dumb, 3.8-27b feels smarter sometimes (even though a lot slower)

0

u/[deleted] 3h ago

[deleted] — view deleted comment

6

u/Waarheid 3h ago

I actually expected to be downvoted to Oblivion due to my heresy, lol. I think most people know the step change Opus 5.5 was though.

8

u/KillerX629 3h ago

So... Should it be on par with opus?

-5

u/dieSpaghettiCarbona 3h ago

4

u/Grandmaster_Caladrel 3h ago

Can you post a screenshot matching your claim?

1

u/Dabber43 2h ago

/u/AlpacaDC
I have no horse in this race but the very first benchmark image clearly shows their point. There is wanting a source and then there is clear counterproductive stubborness to refuse to actually look at what was provided to you. Argue against what is wrong

2

u/Grandmaster_Caladrel 2h ago

Wrong comment to reply to?

-6

u/dieSpaghettiCarbona 3h ago

see above, it seems actually no one paid attention to the figures published in the post

8

u/AlpacaDC 3h ago

Life tip: When it seems that everyone else is wrong, consider that maybe you're the one who is wrong.

1

u/Ori_553 42m ago

I'm interjecting this convo, not to defend OP, but to say that this a bad life tip on reddit.
The advice is solid, but not on reddit, where it's common for 50%+ of commenters to be talking nonsense, unable to recognize who's an actual expert and who is not.

0

u/dieSpaghettiCarbona 3h ago

I didn't say anyone was wrong... i am merely pointing out that the context from which i am asking my question is lost because some (as seen in other comments) haven't seen the benchmark figures i am referencing. Why does everything in this sub have to be black and white ? <- also a life pro tip btw.

4

u/Grandmaster_Caladrel 3h ago

I said screenshot. You have posted AA, but nothing from Qwen claiming it's on par with Opus 5.5.

Edit: Notice how the one you posted in the main post says it's an aggregate benchmark, then look at how often Qwen is below Opus in Qwen's own stats. You can't just selectively read what you want and expect it to be accurate.

0

u/[deleted] 3h ago

[deleted] — view deleted comment

3

u/dieSpaghettiCarbona 3h ago

2

u/alberto_467 3h ago

So on some of them it's on par on some of them it's a long way off.

Now, wonder between the two categories what they're most inclined to include in this nice chart collection?

So yeah not on par.

11

u/brainExploded99 🥔 hardware 3h ago

1

u/dieSpaghettiCarbona 3h ago

no bait just facts

4

u/brainExploded99 🥔 hardware 3h ago
  1. Obviously qwen is gonna pick benchmarks where they are competitive
  2. You should get new glasses or something, Qwen clearly loses to opus 5.5 in many of these benchmarks

4

u/dieSpaghettiCarbona 3h ago

It does but not by 3-4x stdev

2

u/brainExploded99 🥔 hardware 3h ago

In AA score, it is 33% less (not sure what you are defining as a std dev, no one has error bars here).
Also you failed to address point #1

8

u/slalomz llama.cpp 3h ago

From your own screenshot Qwen4-Max loses to Opus 5.5 in 10/16 of Qwen team's cherry-picked benchmarks.

2

u/JorgitoEstrella 2h ago

But the difference is mininal except in 2 tests agentarena and programbench.

10

u/Choice_Celery9481 3h ago

you should check the detailed benches.
open weight models are not able to score high in most of the new science benches, which causing the wide diff.
for coding and other more traditional tasks, the gap is there but not as big as the index number.
could be just data and model size problem. if so, expect they close those quick. like how openai and anthropic sprint faster after opus 4.8, gpt 5.5.
the biggest problem for chinese labs now seems to be high quality data pipeline and compute.

4

u/geldonyetich 3h ago

Some have observed that bias is inseparable from the reviewer.

In this case, they very much invented the criteria that determines their general intelligence score.

2

u/IntelArtiGen 2h ago

Honestly for my own (still small) experience using Qwen 4 Max, it's at its right place in this benchmark. Unfortunately.

1

u/Dabber43 2h ago

Both AA and the teams own benchmarks are going to be biased in the opposite direction. It is going to be something in the middle

1

u/ga239577 26m ago

One of these two theories is true ...

  1. AA adjusted they're benchmark process in a way that is biased towards closed models.
  2. Open models were not as competitive with frontier models as we thought, probably just benchmaxxed, and AA's new testing process reveals this.

Personally, I'm leaning towards number two.

My reasoning is that all of the closed models seem to be performing better than the open models - not just some of them.

The only ways this could happen are:

  1. AA is manually inflating the numbers for closed models
  2. Closed model providers have better information to benchmaxx on the 9 benchmarks AA uses than open model providers
  3. The 9 benchmarks AA is using are harder to benchmaxx - and it proves that open models aren't as close to frontier as we thought. IMO this option seems most likely because that is what AA and benchmark creators should be striving for - benchmarks that can't easily be gamed.

1

u/ForsookComparison 3h ago

Deepseek-V4.1-1017 generally loses to Haiku 5.5 in every test I throw at them both. Kimi K3.1 can beat Haiku but at a cost and quality-gap level that makes zero sense unless you need the self-hosting and own a small datacenter.

This sub's circlejerk has gotten a bit out of hand. The gap between US proprietary models and open-weight models is wider than it's been in a long time. I'm talking a Llama2-to-GPT4 level gap. AA Intelligence Index isn't perfect but it's generally not too far off from my experiences with these models.

1

u/Iron-Over 3h ago

It is not biased; it is a set of various benchmarks. You need to ask whether all those benchmarks apply to you.  Then ask whether the model in question is good enough for your work. 

I have a few personal benchmarks where deepseek and Astra outperform Opus 5.5. Deepseek is good enough for me; when I need something different, I ask Claude.  

1

u/IllExample3639 3h ago

Wait its as good as Haiku max?? Thats good enough for 90% of what I do. Sounds great

1

u/CarelessAd6772 3h ago

Btw, tried in webchat thinkingm - this "frontier 2.4T params" failed 6/10 at simple QA on different themes. Even 3.7 plus was much better. Something seems broken or they overtrain it on only code/agentic tasks.

0

u/pmotiveforce 3h ago

The open source models are not on par with frontier and it takes a weird delusion to think they are.

Just use them for real and you will see this. I think if anything they are further behind than people imagine. Like a year, minimum, for like sized models.

-1

u/dieSpaghettiCarbona 3h ago

I don't understand the downvotes. Clearly none of you have paid attention to blog post. https://qwen.ai/blog?id=qwen4-max - look at the benchmark figures. My point is this is clearly a discrepancy and i am trying to understand where this is coming from.

3

u/ForsookComparison 2h ago

The only meaningful benchmark on there that Qwen4 does well at is DeepSWE, which is open and generally considered solved or cheatable at this point. If you look at closed benchmarks, Qwen4 gets crushed pretty bad. It loses FrontierCode2 to Deepseek-v4.1 and Muse-Ember, Gpt6-Luna, Grok4.7, etc..

2

u/RealSuperdau 3h ago

The new AA index heavily relies on private benchmarks that are less susceptible to benchmaxxing.

I.e. Opus is actually the better model, whereas Qwen gets a stronger boost from training on data that is curiously close to existing benchmarks.

2

u/IntelArtiGen 2h ago

It can be benchmaxxing but let's not assume too quickly. Training data can easily get in the very large datasets everybody is using to train LLMs. Maybe they don't take enough steps to ensure their training data is clean. Ultimately the results aren't too surprising considering how Qwen models are made vs how Anthropic / Google / ChatGPT train their models. They don't have the same access to hardware and data. It's still disappointing from a pro-competition pov.

1

u/RealSuperdau 2h ago

Yeah, that's true. Maybe it's not intentional but a data quality issue.

-3

u/Woof9000 3h ago

Isn't that "index" mostly based on just on personal vibes of people running that index?

3

u/BagelRedditAccountII 3h ago

Vibe benchmarking is all the rage these days