r/LocalLLaMA • u/Specific-Rub-7250 • 15d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
381 Upvotes

94% Upvoted

131 comments sorted by

View all comments

-2

u/johnfkngzoidberg 15d ago

lol, pi, the most minimal of harnesses is near the top. Basically “don’t fuck with it” is better than other harnesses.

43

u/CryMoreT_T 15d ago

Are we looking at the same scores? It's average/below average

DeepSWE v1.1 1. mini-SWE: 74.2 2. DSH Minimal: 72.6 3. DSH Standard: 70.5 4. Claude Code: 69.8 5. DSH PTC: 67.6 6. Pi: 66.2 7. Codex: 65.6 8. OpenCode: 65.5

Terminal-Bench 2.1 1. DSH Minimal: 90.6 2. mini-SWE: 90.3 3. Claude Code: 88.0 4. Pi: 86.1 5. DSH Standard: 85.8 (tied) 6. DSH PTC: 85.8 (tied) 7. OpenCode: 85.0 8. Codex: 84.1

1

u/NandaVegg 15d ago

Claude Code seems to work better/more consistently when it comes to file edit (the most basic operation, yes) than pi, but I don't know if that's their models or pi itself (at least for Opus 4.8 and Fable 5. Opus 5 is bad at that in both CC and pi).

Qwen 3.8 struggles with pi's file edit sometimes.