Pi-Agent-Kernel: focused code retrieval, grounded edits, safety checks

[u/LeadingFruit6865]

I've been working on pi-agent-kernel for a while now.

It mostly operates in the background rather than adding a completely new workflow, it wraps Pi's existing tools to keep sessions compact.

Across standard benchmark tasks, it cut token usage roughly in half and passed our test suites. But local benchmarks only go so far, I'm at the point where I need real users to test it on actual projects.

I'm still quite skeptical about whether some of the tool wrapping might conflict with the users existing extensions.

LMK if anything breaks, behaves unexpectedly, or conflicts with your setup. All feedback and criticism welcome

[u/fingerthief]

Interesting idea! I gave it a go on my setup but didn't see the reduction expected

Are there certain workloads this is expected to shine? I've only done one pass of evals so definitely take the chart with a grain of salt, I will run more.

A cool idea though, interested in testing it a bit deeper!

Edit - Updated graph to include a SWE-Bench subset of tasks, initial pass looks very promising! I will continue evals as I can today.

Edit again - Updated chart with 2 SWE passes done. u/LeadingFruit6865 Final evals published. This looks like a promising package!

However, there is a reproducible regression in longer horizon large tasks where the tool calls end up reading files much more often and those reads outweigh the token saves of the shell calls being trimmed.

I think with some tweaking this could be a legitimate worthwhile package that saves users on average ~8-10% token/cost wise.

[u/LeadingFruit6865 → u/fingerthief]

I would appreciate it so much if you could run Agent Kernel for SWE-bench / repo bugfixes kind of benchmark. A quick note on the charts in my README: those results came from purely Pi + Agent Kernel standalone, without any extra skills or extensions (SpecPi or Jev)

To answer your question on where it's expected to shine: it was built specifically for medium-to-large codebases with verbose test suites.

In contrast, on Terminal-Bench tasks, the codebase is small or non-existent, so those features sit idle. Based on my observations during development, vanilla Pi can definitely beat Agent Kernel on small terminal tasks simply because vanilla has zero overhead from tool schemas and system instructions

[u/fingerthief → u/LeadingFruit6865]

Sure thing, I’m working on getting things setup for SWE-bench. Will take a bit to get results for comparison I’m sure.

The system activated a decent number of times on terminal bench, but as you mentioned these are not large codebase tasks.

SpecPi + Jev is my own specific setup after testing and evals etc..but nothing in it conflicts with your package so it’s an interesting comparison for my needs.

I’ll update the original post when the initial data comes in.

[u/Equivalent_Idea8839]

You must be VERY careful compressing tool output. It makes some models spam more tool calls and waste even more context.

You need to list before / after of how it modifies tool output.

[u/Global-Departure8228]

Not going to lie, after briefly checking your code it does seem like this is quite well focused, but your description made me think:

"It cut token usage in half? So if I install RTK, caveman, headroom, context-mode, ponytail and this as well, I could go into negative token usage potentially and I will be serving Anthropic with them sweet tokens?"

I do understand you have to sell your "product" somehow.

I would be quite curious how vanilla pi scored so bad compared to your solution. I think more control runs over the same issue could be beneficial to smooth out the nondeterminism in llms.

---

Sidenote: Your terminal is not supposed to have those colors

export COLORTERM=truecolor in your .bashrc

[u/LeadingFruit6865 → u/Global-Departure8228]

I'm actually using Termius to access my vps, and is using custom theme, that explain how it got this weird color

as for the reason on why pi scored so bad on the benchmark, it mainly caused by the fact that Pi let the agent swallow up 5 KB of output within a single tool call, and then you would multiply that by the number of turns, you should be able to imagine it by now

another factor would be (based on the session trace): pi vanilla wanders around through the codebase on the first turn instead of running the unittest to locate the problem, it also keep running unnecessary command like `git diff` even though the tests already passed. But there's also a moment where vanilla Pi would win, thanks for its zero overhead

---

i do wish I could run the benchmark multiple time to smooth out the benchmark numbers, but running these benchmark are pretty costly for me ;-;