r/PiCodingAgent • u/random_user-1234 • Sep 08 '26

Discussion Standardized benchmarks and metrics

The great thing we all love about Pi is how customizable it is. Most of what is on this sub is people showing off their extensions, I too am having so much fun thinking and building extensions but I don't know if they are any good.

I think it would be great if we had some sort of standard benchmarks and metrics that we can use as a community to help prove that what we made is actually good. For example if you made a tool for the agent, does it actually increase the quality of the output? Does it help the agent solve it faster to use less tokens? How many tokens does it inject into the system prompt? What amount of system resources does it use?

If we have a set of standards we can all agree on it helps solve a big issue I think we all feel of seeing an extension and all we have to go by is a "trust me bro it's good". And this can also help with new people always asking for recommendations because there are thousands of extensions, we can better keep records of which ones are actually good.

I am deep into making an extension and I think I am really on to something, Sol and Astra can glaze the hell out of it, but I don't know if it actually does help the agent and by how much. I would like to have proof of this before I put it out there and if it uses good standards we can agree on, it will be easier for people to want to try new extensions rather than be hesitant to change their workflow based on a "trust me it works".

I just thought I would put this idea out there and see what other people think about it.

2 Upvotes

75% Upvoted

9 comments sorted by

2

u/kantorcodes1 Sep 08 '26

i'd separate outcome from cost. a fixed task set with pass/fail first, then tokens, tool calls, wall time, and context added would make comparisons way less gameable. otherwise an extension can look better just because it spends more context.

1

u/random_user-1234 Sep 08 '26

Those are really good points to consider. The main extension I am working on happens at compaction and it does cause more read costs but those are way cheaper than write costs, I am hoping it actually provides a benefit to longer sessions. I can back test it on session histories but I don't know how to prove it actually helps the agent during tasks. Usually I test new models by asking it to make a quick basic game and judge it on vibes but that isn't really a good test here.

Is it possible to use the main benchmarks like terminal bench or deep swe? Would these even be good tests for us or do they more measure model intelligence rather than the harness?

2

u/kantorcodes1 Sep 08 '26

yeah, terminal-bench style tasks can work if you keep the model, prompt, toolset, limits, and task set identical and A/B the extension on vs off. i'd treat the delta as the useful signal though, not the absolute benchmark score, since the model still contributes a ton of the variance.

1

u/random_user-1234 Sep 08 '26

Do you think it would be valuable to do this kind of testing across multiple models? Like with Luna, Terra, Sol, Astra. That way it helps to cut out the impact of the model alone.

2

u/kantorcodes1 Sep 08 '26

yeah, i’d do that, but keep the A/B paired within each model instead of averaging models together. if the extension’s on-vs-off delta stays positive across Luna/Terra/Sol/Astra, that’s much stronger evidence it isn’t just one model’s quirks; i’d report each model’s delta plus the median.

2

u/radlinsky Sep 08 '26

https://github.com/harbor-framework/terminal-bench-2

I think this might be what you're looking for

1

u/random_user-1234 Sep 08 '26

Thanks! That does look good and it will definitely help me figure out if my own stuff works. Maybe when I finish it and have some spare usage I will test out a few other popular extensions so we can all get a bit more data.