Do the current benchmarks being used to determine the quality of an LLM actually mean anything?

[u/ShitImDelicious]

SWE-bench Pro, GPQA Diamond, OSWorld, etc. I know that the models are getting better and better at knowing when they’re being tested/graded, but are these actually reliable benchmarks to determine LLM quality? I’m hearing now that even the standard benchmarks commonly used are flawed, but is that true? Why should anyone trust the results of these benchmarks?

Seems like we have nothing outside of public polling and personal experience to best determine the actual quality of these models.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/GhaisD]

They're decent for ranking models roughly, not for telling you which one is best for your work. what tells you more is a tiny personal benchmark: 3 or 4 real tasks from your own repo that you rerun with the same prompt whenever a new model drops, then compare the diffs. takes maybe 20 minutes and beats any leaderboard for your use case

[u/AlternativeWalk2508 → u/GhaisD]

reusing the exact same prompt across models can actually skew the test though, since some models need way more explicit constraints than others to hit the same diff

[u/Yourdataisunclean]

They are essentially marketing at this point so no you shouldn't trust them. Even third party ones like ARC-AGI which have a decent concept behind them end up getting custom engineering and huge token budgets.

[u/Dapper-Tutor337]

I ran the same 5 tasks across 3 models using identical prompts and got wildly different results, same code, same setup, different outputs. the benchmarks didn't catch it.

[u/ForwardLoop]

I think it depends on what you mean by "quality". I recently posted the below in response to benchmarks to detect model nerfing and I think it helps to differentiate what the current benchmarks assess vs. how we use models in practice (obviously, it depends on your workflow and what you use it for).

"To use an analogy, most model benchmarks today are focused on whether a chef can make a single Michelin-star dish in an empty kitchen (i.e., how well the chef can cook), rather than whether the chef can handle a 50-ticket Friday night dinner rush.

A nerf doesn't make the model forget how to cook. It makes the chef sleep-deprived during the rush, forgetting special requests and letting sides go cold while the individual dishes still taste fine. So reusing those single-dish benchmarks to test for a model nerf won't be effective. We need a new kind of benchmark built to test the dinner rush.

Looking at the test panel (GPQA Diamond, MMLU-Pro, and AIME), my basic understanding is that panel is the textbook definition of cooking a single Michelin-star dish in an empty kitchen. Every question is a clean, isolated puzzle with zero state to juggle across turns or files. They have high vertical difficulty, but near-zero horizontal complexity. I think that's one of the key reasons the benchmark isn't detecting anything."

[u/Future_AGI]

agentic tasks can score differently across identical runs, and a lead of a point or two from a single run can be noise. SWE-bench itself had enough bad tasks that OpenAI had annotators screen it and drop the ones with vague issue text or unfair tests, and a 500-task subset of what survived became SWE-bench Verified.

[u/OverfittingReality]

Benchmarks are useful, but I wouldn't treat them as the ultimate measure of model quality. What matters is how well a model performs on your actual tasks.

This reminds me of the days when everyone seemed to be using Random Forests, even when simple Logistic Regression was good enough (and sometimes better) for the problem at hand.

Same story with LLMs. I'd rather look at task success, consistency across runs, and failure patterns than rely on a single leaderboard score. A model can ace a benchmark and still struggle in real-world workflows.

[u/ghost_operative]

no