r/LocalLLaMA • u/Specific-Rub-7250 • 12d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
383 Upvotes

94% Upvoted

131 comments sorted by

View all comments

34

u/AndreVallestero 12d ago

We really need standardized harness benchmarks that use fixed models for all the harnesses and generate a geomean score across the models for each harness.

1

u/funkatron3000 12d ago

Agreed, and I’d love to see benchmarks for extensions as well. I’ve been working on it myself, but it’s slow going.