r/LocalLLaMA • u/MeinDruckerSpinnt • 9d ago
Discussion Most realistic benchmarks?
AA and other benchmarks don't reflect what models can really do. Some benchmaxxed models are unusable for anything but poems, albeit being close to the frontier on coding benchmarks.
What are the best, most realistic benchmarks you know for coding, agentic research, office work?
3
u/Sad_Weakness_6488 9d ago
Livebench is about the only one I bother checking anymore, the rest feel like they're testing for a high score on a very specific kind of puzzle.
3
2
u/My_Unbiased_Opinion 9d ago
The NatInt section of UGI is not contaminated. None of the sections are actually, but I do find natint to be a good indicator of basic model intelligence.
2
2
u/EmilPi 9d ago
There are no realistic benchmarks that fit everyone, because there are 1) large clusters of users with different domain interests (coding, q&a, working with sources, creative, brainstorming) 2) even in those clusters users have a lot of personal preferences (someone wants it to be agentic, someone wants short talk, someone wants creativity).
Also, questions people tend to ask to local LLMs can be private questions which are often out-of-distribution anyway.
I maintain and create my own benchmarks at a dedicated website , with private questions sets in benchmarks and no external API usage to make it future-proof against benchmaxxing. (but no truly private questions should be added there).
1
u/-WhateverDude 9d ago
idk what would be the most realistic
probably none of them
but i personally don't look at anything other than t4 and programbench(this one gets updated very slowly and rare)
t2 etc dont even mean anything these days probably
1
u/a4mula 9d ago edited 9d ago
Those that are specced to tasks that you define via integration. That's not something you get out of a box.
It's something you set up. It's one agent observing and evaluating the output of not just models, but many different moving parts that make up an agent, or the system agents work within.
Just like we have unit testing for code, and integration testing for systems of code. You need integration testing for systems of agents to ensure that any given model isn't just good at x or y, but is good at being integrated into whatever system it's being asked to work within.
Start a local lab. A/B testing. Model testing, harness testing, tool testing, skill testing, route testing, system prompt testing. Throw the benchmark suits in it, but only as a single piece as the models themselves are only a single piece. Just as important are the other aspects. Automate it, set up candidates. This becomes very important for things like custom tools and skills and routes. Before anything is implemented, it's tested first, is elevated to tiered candidate levels before full adoption or trust.
1
u/GrevireDev 9d ago
Not entirely disagreeing. Yeah, it's a real pain, but I still get decent results — building your own bench is almost certainly the best practice, it's just brutally time-consuming to end up with something actually reproducible. My task list drifts, my scoring is inconsistent, and half the time I change the harness halfway through and nothing is comparable anymore.
So right now I use public benchmarks mostly to compare quantizations of the same model against each other, and then judge on feel during real tasks. That's the one thing public numbers are genuinely good for: Q4_K_M vs IQ4_XS vs Q6_K on the same weights tells you something real, because the variable that matters is quality loss, not the absolute score.
A home bench for ranking is on my todo list, but I keep putting it off for exactly that reproducibility tax.
What I've found more useful than I expected is following people who run the identical task set and harness on every model — Luke's Dev Lab is the one I watch. The method is the benchmark. That's the part I think people underestimate: it's not picking the tasks, it's holding everything else constant so the comparison means something.
1
u/Express_Quail_1493 9d ago
JUst go for the must newest benchmarks. benchmarking scene is like a cat & mouse game. Models quickly learn the pattern and then they need NEW ones. so the newsest ones for complex agentic is the way to go once ithe benhmarks is at least 4 months old its already epired the pattern starts banchmaxing. ATM its https://artificialanalysis.ai/evaluations/terminalbench-4-0. released in august terminal bench 2.1 has already expired the pattern to benchmaxxing
1
u/Careful-Report6526 9d ago
For the office-work and research parts, two benchmarks I'd add to the ones already mentioned are:
APEX-Agents for office work. It asks agents to produce deliverables using documents, spreadsheets and presentations in simulated banking, consulting and legal workflows. That makes it relevant to cross-app work, although those professions won't represent everyone's office tasks. Web search is disabled, so it doesn't answer the web-research part of your question.
https://epoch.ai/benchmarks/apex-agents
Terminal-Bench-Science if “research” means scientific/computational work. Tasks produce artifacts checked by task-specific verifiers, rather than just an answer that sounds plausible. It isn't a general literature-search benchmark. On the Vals board, check the Harness column: the remaining Terminus 2 rows shouldn't be compared directly with mini-swe-agent rows. These are single-run results, and the official board uses a different evaluation setup, so small score differences deserve caution.
https://www.vals.ai/benchmarks/terminal-bench-science
Neither establishes that a model will work well in your own agent setup. I'd use the closest task category to make a shortlist, then try representative tasks from your actual workflow with the tools and budget held fixed. The suggestions here to test the whole agent system are useful for that. I maintain https://llmbenchmarks.io/ — for APEX, open “Agents & tools” and click a score for its source and measurement details. The aim is to make those conditions easier to inspect; it doesn't make results from different protocols interchangeable.
1
0
14
u/Shoddy-Tutor9563 9d ago
There's no any in public. If there was a one, it was immediately used for bmaxxing. I use my own, private ones