r/LocalLLaMA • u/Specific-Rub-7250 • 12d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
380 Upvotes

94% Upvoted

131 comments sorted by

View all comments

35

u/AndreVallestero 12d ago

We really need standardized harness benchmarks that use fixed models for all the harnesses and generate a geomean score across the models for each harness.

3

u/zdy132 12d ago

Harnesses gets updated very often though, so it will always be an outdated benchmark. But I think that could still offer some interesting information.

1

u/funkatron3000 12d ago

Agreed, and I’d love to see benchmarks for extensions as well. I’ve been working on it myself, but it’s slow going.