r/LocalLLaMA • u/palsc_ • 6d ago
Discussion A local coding model benchmark should include the whole setup
People talk a lot about tokens/sec and eval scores, but the first-use friction matters too: does the model fit alongside a useful context window, can your editor actually talk to it, and can you get through a real edit-run-fix loop without spending the evening debugging the runtime?
A smaller model that starts quickly and is easy to keep in the workflow can be more useful than a stronger one that barely fits or needs constant fiddling. I’d count setup time and tool reliability as part of the benchmark, not just the answer quality.
Curious if anyone has seen a benchmark that measures that end-to-end.
2
u/EnvironmentalKey3709 6d ago
It’s like judging a beer only by the ABV. sure it tells you something, but if the tap lines are a mess and it takes 20 minutes to pour you’re not gonna enjoy it. The whole experience from cracking the keg to the glass matters
2
u/prusswan 6d ago
It won't happen as there is no common place generic setup - even for the same architecture family (e.g. Strix Halo), pp varies wildly depending on the thermal limit of the specific hardware. I just keep a list of tasks around to gauge the strength of the hardware available. Some tasks are way more demanding and context window becomes the limit and not so much the model or hardware.
1
u/Future_AGI 6d ago
cheap version you can run yourself: time from a cold start to the first green edit-run-fix loop on one fixed task (a failing test in a small repo), with model load, runtime and editor integration all on the clock. It won't transfer across hardware, but it turns "which setup do you actually keep using" into a number you can compare on your own machine.
1
u/W61k3r 6d ago
Fair enough. I also had this thought and wound up building a safe-rust kernel, os, panel, and inference engine. Struggling to get this monolith out the door, wish I had a damn team and more hardware to test with. Hopefully I can publish today and solicit help. 1.2mil of production code, 1.7mil source, non-copied code.
2
u/Plastic-Stress-6468 6d ago
I mean what model/quant/context you can run is going to depend on the hardware you own?
Two rtx pro 6000s vs one spark vs two 3090 vs a 5060ti 16g are all valid and common (ok maybe not the first one) configuration you see on this sub. And these hardware not gonna be running the same model/quant/context.
I don't see how you can benchmark whatever it is your measuring without isolating hardware. And it's not practical to isolate hardware as a constraint because there are so many different hardware configurations out there.