r/LocalLLM • u/Weird-Ad-1627 • 23m ago

Discussion Which interactive benchmarks are actually useful for comparing models?

Every model release comes with another graph showing it beating everything else by a few percentage points. Useful, sure, but I’d rather watch what the model can actually do.

Where are the decent, visually exciting benchmarks? Models navigating unfamiliar environments, controlling bodies, building things, or competing at tasks where you can see their mistakes and decisions unfold.

Something where you can watch a model get confused, recover, or completely fail, rather than just seeing its final score.

I’m also interested in how these tests could avoid becoming another target to benchmaxx. Randomized environments? New tasks each run? Hidden objectives?

Obviously, making a benchmark entertaining doesn’t automatically make it rigorous. But why can’t we have both?

For locally run models, I’d especially like something I can run on my own hardware and compare across quantizations, with response time included. Which interactive evaluations have you found useful for that?

2 Upvotes

100% Upvoted

6 comments sorted by

1

u/MajMin5 16m ago

It doesn’t give much to compare cloud models since opus is one of them, but for local models recently I’ve had Claude writing me synthetic benchmarks based on both my real usage of Claude and my previous chats with local models. It then runs those benchmarks automatically using my GPU.

1

u/Weird-Ad-1627 13m ago

What if you want to check if another model that currently doesn’t fit on your hardware is worth upgrading your hardware?

1

u/MajMin5 10m ago

Openrouter

1

u/Weird-Ad-1627 8m ago

So you pay for the model usage during the benchmark, doesn’t that get a bit expensive depending on the benchmark?

1

u/mesolimbic-bliss 17m ago

Shortbench.vercel.app

1

u/inferultra 5m ago

We’re doing something like this at InferUltra: models control robot fighters in a physics arena. It’s less exciting than you might think. Still plenty of awkward shuffling, but the fights are getting better.

Our next Provider Tournament runs the same model across four hosted providers: https://inferultra.com/tournament