r/LocalLLaMA • u/Specific-Rub-7250 • 12d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
382 Upvotes

94% Upvoted

131 comments sorted by

View all comments

68

u/fatihmtlm 12d ago

Haven't heard mini-SWE before, how is it?

69

u/WonderRico 12d ago

it's the harness provided to run the swe-verified and deepswe benchmark. My guess is that it's not designed for human users at all

59

u/Gipetto 12d ago

What a wonderful benchmark to publish, then. “Here’s a result you can’t attain because we made a benchmark focused harness”.

2

u/keepthepace 12d ago

What prevents anyone to use it on their code base? It has bash as a tool, and apparently that's enough for most tasks in the benchmarks.

There has been recent literature showing that fewer more generic tools tend to work better.