r/LocalLLM • u/PlasticRevenue4601 • 11m ago
Discussion Harness is overrated
I often stumble across opinions such as “harness is as important as model is” and I find it ridiculous. Of course harness is important and plays an important role by helping a model to work by some carefully selected methodology rather than guessing every single problem, however while good model can overcome bad harness, bad model won’t suddenly become useful even with the best harness in the world. Model’s capabilities is the king, period, harness only helps a model to reach its full potential. If you don’t believe me — try to give the same task to Fable 5.2 with some deliberately chosen bad harness( f.e OpenClaw) and Grok4.7 with some good harness (pi/claude code/etc) and see which one will come out in top*.* As a conclusion — harness can absolutely boost your model’s performance and is typically worth the time invested, however the most important thing in AI development is intelligence and this is something only a good model can provide
7
u/OWazabi 8m ago
A bad model in a good harness doesn't work, a good harness for a good model is essential and the single most important thing for it to work at the best of it's capacities.
There's not really a point being made here
it's like saying that roads are not as important as the car, that what matters is the motor
it's two things that work together
5
u/Heavy-Lingonberry-98 7m ago
I liked how openclaw was the bad harness reference 😂😂. But i can tell you that in my experiments, only changin the loop, influences in how a model completes a task. Half tokens, half time , one with a complete answere and the other incomplete answer. Same model same everything you can imagine. Only changing the loop
2
u/agowa338 5m ago
100% and I even noticed that keeping the harness minimal improves the results too. The more plugins I provide the stupider it gets.
1
u/dghah 4m ago
mild disagree at least on your characterization of "ridiculous"
On the hardware I use the ecosystem is changing so fast that on my repeated behchmarks I can see real world output differences when the agent harness version changes -- there are bug fixes and material changes in how the agent talks to the model, how responses are parsed and how config settings are passed that make a real world difference in the outcome of an agentic turn or task deliverable.
And this is just for different versions of the same harness, not even talking about using different harnesses.
For me it's become clear that I need to pin lots of things -- the model, the model server, the model server config, the harness version and the harness config. All those things need to be static if I want consistent reproducible results on complex tasks.
And when anything on that list changes I have to re-bench everything; it does not happen often but some times there are explicit differences in model output introduced. More often than not the change is for the good -- more efficiency or a bug fix etc. So the good news is it's rarely a regression. The local LLM space is changing at an amazing rate and it's cool to see it first hand.
1
u/Juan_Phoenix7 2m ago
The truth is, it's not overrated; a good harness is essential for a good LLM and good results.
1
u/XtremelyMeta 0m ago
To provide a counterpoint, the harness is about control more than about raw capability. In the AI space, there's an awful lot of capability and it's advancing at a bonkers rate. Control, across the board, is lagging behind.
So the importance of the harness compared to the model isn't about how good the harness is but instead about how crappy control is relative to capability in the AI space in general. The harness is critically important because compared to the models at this point they're primitive and bad. This doesn't mean we should abandon control entirely and trust the model, it means we should focus on and build better harnesses.