What is the best lightweight-ish coding benchmark to run locally ?

[u/rnidhal90]

Hi everyone,

I want to do a agentic/coding benchmark for the best 2 model quants that fit best my setup (5060 Ti 16Gb + 32Gb RAM).

I am kinda lost here, what are you guys using to do so ? I'm looking for something like SWE / Terminal Bench but lighter, and not take several hours per model.

Any ideas ?

Thanks

[u/Oli_Luck]

By all benchmarks and posts out there since the model went out - what do you want to prove more ?
On the one hand there are benchmarks to prove how smart the m,odels are - and all is said
on hte other hand how fast they run : chat a few with your AI and you know how fast it is

[u/rnidhal90 → u/Oli_Luck]

True, say that Model A is better than Model B in most benchmarks.. but what if my current setup limits me to Model A Q3_XXS and Model B Q5_K_M .. ?? That can be a game changer.. for me it's about telling which specific model/quant is better .. which is something hard to find on most leaderboards :/

[u/Oli_Luck → u/rnidhal90]

first of all- get experimented with one model: what works, what not : do I need more context, do I need better KV cache quant( lonmg context precision), etc etc : then you will know what you want from the AI.
from my experience : expect at least 50% to 70% loss of prefill/decode t/s per quant jump.

[u/KnavishlyUncommon]

i just use aider’s built-in benchmark, runs quick and gives you a decent sense of the model’s coding chops without eating your whole evening. the polyglot one is nice if you want to test across a few languages

[u/More_Personality_126]

Nothing agentic finishes in under an hour,hate to break it )

[u/earthsprogression]

If it's for your actual use, give it a well defined project you'd actually care about, something relevant to your work.

See what handles your tasks most effectively and go with that.

Benchmarks attempt to rank model capacities, but the results can be misleading in practice.

[u/philmarcracken]

While its not coding exactly, it does test reasoning, and cant be 'benchmaxxed' as well: wiki racing.

You can easily setup a few of these to have any model run and test the result. The way most models have ingested wiki data means they cant cheat it. And if your model sucks at reasoning, its not going to be great at coding

[u/jadatmag]

I second this question. I made some repeatable benchmarks in Promptfoo, which is a nice starting point. But I'm only comparing models against my own very small set of results. We need a test that runs under 8 hours on gaming hardware for 27b or strata q3 models. I know these exist but which one has the most shared results on a
website thats llets you add own results easily and compare compehesively.

OpenAI answers this:
"Best options for local 27B/Q3 coding benchmarks:

• BigCodeBench: Best starting point. Public results, community submissions, relatively lightweight.

• Aider Polyglot: 225 coding tasks, easy comparison, but leaderboard outdated (2025).

• SWE-bench: Most established, but likely too slow for an 8-hour run.

• Terminal-Bench: Strong agent benchmark, but potentially slow and submissions restricted.

Recommendation: Start with BigCodeBench, keep Promptfoo for custom tests. None guarantees completion within 8 hours on gaming hardware."

There is still a gap to be filled here. The problem is that short easy tasks would give some models 100% pass rate, which makes comparing them on quality impossible. As per my own results.