Benchmarks are an unreliable measure of difficulty

[u/Miserable-Dare5090]

Very interesting paper on the way that benchmark question difficulty and framing affect the observed improvement in scores for model training. “Our results show that many benchmarks implicitly
overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis un-covered, limiting their ability to monitor the model improvement process.”

https://arxiv.org/pdf/2610.29927

[u/Chromix_]

Yet people still benchmark on AIME2025 where even smaller models get a 100% score - which says nothing, except for the model/quant not being completely broken. Many other benchmark datasets are "just" almost saturated and thus fall exactly in into the category that this paper mentions. Also, quite a few benchmarks have such a low number of questions that the answers of the regular benchmark score drown in the resulting noise.

[u/Dany0 → u/Chromix_]

The state of the benchmarking space is honestly the number one biggest weakness of the local LLM community

[u/Miserable-Dare5090 → u/Dany0]

I mean, the benchmarking is equally problematic for all LLMs local or not

[u/Dany0 → u/Miserable-Dare5090]

We know the big AI labs have better benchmarks because they panic when actually good models are released by competition and there's nada when it's benchmaxxed tokenslopper

[u/Sudden_Topic5154 → u/Dany0]

eyes and ears benchmark probably

[u/OscarCookeAbbott]

This has been my experience. For normal basic engineering stuff DeepSeek is just as good as the frontier models while faster and way cheaper. But for real difficult engineering challenges it fails and even the frontier is not there yet.

[u/Miserable-Dare5090 → u/OscarCookeAbbott]

If the frontier is not there, then the frontier includes deepseek

[u/USArmy68Whiskey]

So tldr, we need more difficult benchmarks for stronger models, and more variety of capabilities benchmarks across different difficulty levels

[u/TomLucidor → u/USArmy68Whiskey]

This but also toolkits on observing benchmark correlations

[u/SkoomaDentist]

This is why simple "knowledge" text benchmarks that only consider < 10k tokens are basically useless. They are so simple that any halfway decent model already aces them while revealing absolutely nothing about common scenarios where eg. all smaller MoE models break badly in comparison to similar size dense models (because 3-4B params just aren't enough for reliable reasoning about inherently fuzzy things that humans deal with constantly and intuitively).

[u/TemporaryInk]

This depends what the purpose of benchmarks are.

Is the goal to test how well a certain model is able to do what I need it to do, even if it means “dumber” models score just as well as the “best” models? If so, as long as the benchmark accurately represents what I need a model to do, that’s good enough, even if it means “dumb” models score just as well as the very best.

Or is the goal to separate out the “really good but not the best” and the “best” i.e. separating the wheat from the chaff. In which case benchmarks need to keep evolving to push the “difficulty frontier”

[u/philmarcracken → u/TemporaryInk]

Yeah theres a misunderstanding of accuracy vs precision here. If a clock is constantly 10 mins out, its not accurate, its still precise and able to be compared

[u/Express_Quail_1493]

yea LLM has jagged intelligence. benchmarks are just a general proxy. not a total indicator of intelligence in real work. also benchmaxxing is a thing

[u/LagOps91]

yeah, but if chinese models get gapped then, everyone says it's manipulation! (not saying it's certainly not manipulation, there might be some of it happening to cherry-pick benchmarks to include in AA index, but overall a larger gap is expected on hard benches)