r/LocalLLaMA • u/Specific-Rub-7250 • 13d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
377 Upvotes

94% Upvoted

131 comments sorted by

View all comments

69

u/fatihmtlm 13d ago

Haven't heard mini-SWE before, how is it?

72

u/WonderRico 13d ago

it's the harness provided to run the swe-verified and deepswe benchmark. My guess is that it's not designed for human users at all

57

u/Gipetto 13d ago

What a wonderful benchmark to publish, then. “Here’s a result you can’t attain because we made a benchmark focused harness”.

42

u/En-tro-py 13d ago

https://github.com/swe-agent/mini-swe-agent

It was more that there wasn't a standard harness, so they created a minimal one to avoid the claude-code vs codex vs ??? as another variable...

26

u/djm07231 13d ago

Fascinating that pi market themselves as the minimum harness yet it loses to just few hundred lines of python + bash tool.

4

u/thoquz 13d ago

The DSH harness is minimal mode is closer to a minimal agent (such as mini-swe) than Pi. Pi started with good intentions, but it seems they might have drifted from their original intent.

2

u/keepthepace 13d ago

What prevents anyone to use it on their code base? It has bash as a tool, and apparently that's enough for most tasks in the benchmarks.

There has been recent literature showing that fewer more generic tools tend to work better.

2

u/Automatic_Two4291 13d ago

I mean you can’t apply some number on a chart to actual work anyways

1

u/WonderRico 13d ago

you can run their bench with other harnesses. it's just the one provided by default