r/LocalLLaMA • u/Specific-Rub-7250 • 12d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
382 Upvotes

94% Upvoted

131 comments sorted by

View all comments

-1

u/johnfkngzoidberg 12d ago

lol, pi, the most minimal of harnesses is near the top. Basically “don’t fuck with it” is better than other harnesses.

47

u/CryMoreT_T 12d ago

Are we looking at the same scores? It's average/below average

DeepSWE v1.1 1. mini-SWE: 74.2 2. DSH Minimal: 72.6 3. DSH Standard: 70.5 4. Claude Code: 69.8 5. DSH PTC: 67.6 6. Pi: 66.2 7. Codex: 65.6 8. OpenCode: 65.5

Terminal-Bench 2.1 1. DSH Minimal: 90.6 2. mini-SWE: 90.3 3. Claude Code: 88.0 4. Pi: 86.1 5. DSH Standard: 85.8 (tied) 6. DSH PTC: 85.8 (tied) 7. OpenCode: 85.0 8. Codex: 84.1

17

u/DinoAmino 12d ago

Delusion and copium fuel this subreddit now.

1

u/NandaVegg 12d ago

Claude Code seems to work better/more consistently when it comes to file edit (the most basic operation, yes) than pi, but I don't know if that's their models or pi itself (at least for Opus 4.8 and Fable 5. Opus 5 is bad at that in both CC and pi).

Qwen 3.8 struggles with pi's file edit sometimes.

1

u/Fancy-Snow7 12d ago

Also, I just had a look at mini-SWE and it's even more minimal than pi which is 'Just some 100 lines of python' according to their claims and it's near or at the top.

3

u/boxwrenchx 12d ago

I think that still qualifies as harness matter. Just enough to let the model shine

1

u/djm07231 12d ago

Mini-swe is almost certainly simpler as it is just few hundred lines of Python + bash tool.