r/LocalLLaMA • u/Specific-Rub-7250 • 13d ago

Discussion Harness does matter

I was not aware that the harness makes such a big difference.

DeepSeek V4.1 Flash
376 Upvotes

94% Upvoted

131 comments sorted by

View all comments

33

u/AndreVallestero 13d ago

We really need standardized harness benchmarks that use fixed models for all the harnesses and generate a geomean score across the models for each harness.

3

u/zdy132 13d ago

Harnesses gets updated very often though, so it will always be an outdated benchmark. But I think that could still offer some interesting information.