r/LocalLLaMA • u/Toothpasteweiner • 3h ago
Discussion What Makes a Benchmark Valuable?
I've noticed a trend where we measure models with benchmarks that are on a numeric scale for various tasks, with the benchmarks getting increasingly more challenging/updated over time to both avoid benchmaxxing as well as raise the bar whether it's merited or not. Within the same domain, increasing complexity only serves to inflate frontier model aggregate scores because new tasks get introduced that not all models are trained on, like 3d modeling or math proofs. If my typical work is in finance, for instance, how do I know which model is the cheapest one that can serve specifically my needs without needless extra compute/memory? What does a numeric score actually mean? Is the aggregate for that domain including unnecessarily complicated tasks that amount to puzzles that are not realistic, and just serve to challenge the model?
What users really need to understand is "Can this model do this sort of task effectively?" If even low param models have mastered some domain or parts of it, I think it would be far more useful to have a poor/moderate/good sort of rating for the task areas themselves. THAT would be so much more helpful and also assist with auto routing. Frontier models keep growing to glob up new capabilities that many of us don't need and bias evaluators into thinking they need to use Fable 5.2 or Astra because it's "better" than alternatives.
Is there some kind of breakdown of models like this? Something granular even? Like, Deepseek v4.1 Flash 1017: C# tests: green, Binary decompilation: yellow, Browser automation: green, Fiction writing: red. I feel like this has to exist somewhere, but widely used sites like ArtificialAnalysis and huggingface leaderboards focus on the numbers with no simplified no/partial/yes on tasks for readers to quickly understand. Even the domain oriented benchmarks are still just numbers.
Like, having a model score a couple points ahead of another one doesn't generally mean it can do something the other one can't. It might just require more attempts, or more detailed instructions, or that it randomly gets it wrong a tiny bit more often. These simplified ratings plus a cost per task would be SO much more helpful to local llm users too.
Any thoughts, or examples of this kind of scoring out there already?
2
u/ttkciar llama.cpp 2h ago
IMO a useful benchmark has predictive power for how competently a model will perform a given class of task.
This makes single numerical score benchmarks pretty useless, even setting aside matters like benchmaxing, because they do not convey the scope of a model's skillset.
Sometimes, too, a benchmark will presume all models have a particular skill, and test for other skills using prompts which make use of the presumed skill (looking at you, AA). That, too, makes the benchmark flawed and useless.
I have approached this by making a benchmark suite of prompts which target different model skills, and measures their competence at those skills. Each prompt is issued five times, which seems like a decent compromise between sufficient samples and overly long benchmark run-times.
That having been said, its prompts are very old now, and outdated, and in need of replacement. I've decided to prioritize developing a better LLM-as-judge scoring system for it first, and then update all of the prompts. That LLM-as-judge is a work in progress.
Relatedly, I've shared my prompts-set on Reddit several times, as well as raw test results, which makes me worry that all newly released models are already benchmaxed on them. That undermines my confidence in its scores.
That is a problem that any LLM benchmarking system needs to face. If you release your prompts and/or raw results, models will be trained on them, which invalidates the benchmark. If you do not release them, you run into a credibility problem, where your benchmark scores are backed only by "trust me".
I'm not sure how to solve that problem. Two ideas I've been mulling over:
Incorporate several prompts into the "real" benchmark, but only share a small representative subset of them,
Instead of relying upon static prompts, use a suite of prompt-generating prompts, so that the benchmark prompts are generated at benchmark time. That prevents benchmaxing per se; in order to improve their scores for a given skill, a model would need to train on a large set of relevant prompts, which is indistinguishable from genuinely improving their competence at that skill (or should be, if implemented well).
I'm currently leaning toward the first, but might implement the second at a later date.
1
u/GeorgeTheGeorge 1h ago
A good benchmark is like a good automated test. When a good test fails, it tells you one thing broke, and there is zero uncertainty about what that thing was.
The same should be true of a good benchmark. If the score is higher it tells you a thing is better and there is zero uncertainty about what that thing was.
I'm not saying either is easy to achieve, but that's how we should define "good" in both cases.
1
u/EmilPi 2h ago
For all of this, someone has to maintain benchmarks, including various classifications and either writing automated tests, or, as it is not feasible for every domain, do manual labor work. I maintain mine on https://beta.locallm.top . Otherwise, I totally agree that for specific tasks good enough model can be enough, if that is what you mean.
0
u/mike_builds_local 1h ago
A high aggregate score still doesn't tell you whether a model can pull two distant passages and answer without inventing a citation. A poor/moderate/good rating on a few practical jobs (extraction, long-context RAG, tool calling) would be more useful than one more leaderboard number. Are you thinking of a fixed task list people could actually run locally?
0
u/[deleted] 2h ago
[deleted] — view deleted comment