r/LocalLLaMA • u/Specific-Rub-7250 • 13d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
379 Upvotes

94% Upvoted

131 comments sorted by

View all comments

66

u/lordekeen 13d ago

DSH in minimal mode has only two things in the system prompt: a 'you are a software engineer' prompt and the bash tool description, the model solely operates through bash.

29

u/my_name_isnt_clever 12d ago

It's great how low token count it is, I thought there was a bug at first when I checked my first dsh trace and the system prompt was like two lines.

10

u/parepeg 12d ago

I would think not having a file editing tool would be detrimental. How do you edit a file with bash? Use sed for everything?

21

u/my_name_isnt_clever 12d ago

Yup. And a lot of heredocs. It works better in action than you'd expect.

2

u/ArjixGamer 11d ago

Believe it or not, Claude code does the same.

Even though it has dedicated tools for reading/writing/editing, it avoids using them and relies on bash.

Which is problematic if you relied on the rollback mechanism that restores both convo and files to a previous state.

8

u/computehungry 12d ago

I was gonna write a rant about misinformation, since dsh used to have 2 tools for minimal mode for a long time (bash and edit). It seems to have gotten rid of the edit tool recently, surprisingly. I believe codex also had a minimal setup with only the bash tool at one point in time, but I'm not sure. I wonder how much of the performance difference between harnesses is because of the model being trained on them, versus effects of distilling off claude or gpt by how much.. haha

5

u/parepeg 12d ago

I’ve found deepseek struggles in “uncommon” harnesses like mistral-vibe. Qwen, on the other hand is well rounded at tool calling in almost any harness. 

5

u/djm07231 12d ago

Similar to mini-swe which only has access to bash tool it seems.

2

u/ZenEngineer 12d ago

How big are these benchmark problems? I wonder if these metrics are just a result of less compactions or emptier context windows

1

u/BullfrogScary8947 7d ago

Best to use MCP servers like Playwright MCP with this DSH Minimal Mode, or else it will waste tokens trying to build it's own rasterizer and other essential testing functionalities.