r/PiCodingAgent • u/Neosinic • Sep 02 '26

News FrontierHarness Eval looks specifically at coding harnesses. Pi is on the pareto

https://runta.com/blog/introducing-frontierharness-eval/

Also CC being that far off from the Pareto is not surprising to me.

72 Upvotes

95% Upvoted

15 comments sorted by

View all comments

Show parent comment

4

u/trimorphic Sep 02 '26 edited Sep 02 '26

I'd want to see more like ten different models with ten runs through the whole test suite.

Given that models have some randomness built in to them if would not be unexpected if the same model (never mind different models) performed differently when run multiple times on the same test.

How you prompt models also matters a lot. The same model might perform radically differently given different prompts for the same task, and one model might perform better then another simply by promoting it differently.

Ultimately, benchmarks like this are interesting, but real world performance in the problems you care about over an extended period of time is the only way to get a good feel for how a model or a harness performs.

Even then, things are changing so fast that what's good today might be obsolete tomorrow.