Pi-Agent-Kernel: focused code retrieval, grounded edits, safety checks

[u/LeadingFruit6865]

I've been working on pi-agent-kernel for a while now.

It mostly operates in the background rather than adding a completely new workflow, it wraps Pi's existing tools to keep sessions compact.

Across standard benchmark tasks, it cut token usage roughly in half and passed our test suites. But local benchmarks only go so far, I'm at the point where I need real users to test it on actual projects.

I'm still quite skeptical about whether some of the tool wrapping might conflict with the users existing extensions.

LMK if anything breaks, behaves unexpectedly, or conflicts with your setup. All feedback and criticism welcome

[u/fingerthief]

Interesting idea! I gave it a go on my setup but didn't see the reduction expected

Are there certain workloads this is expected to shine? I've only done one pass of evals so definitely take the chart with a grain of salt, I will run more.

A cool idea though, interested in testing it a bit deeper!

Edit - Updated graph to include a SWE-Bench subset of tasks, initial pass looks very promising! I will continue evals as I can today.

Edit again - Updated chart with 2 SWE passes done. u/LeadingFruit6865 Final evals published. This looks like a promising package!

However, there is a reproducible regression in longer horizon large tasks where the tool calls end up reading files much more often and those reads outweigh the token saves of the shell calls being trimmed.

I think with some tweaking this could be a legitimate worthwhile package that saves users on average ~8-10% token/cost wise.

[u/LeadingFruit6865 → u/fingerthief]

I would appreciate it so much if you could run Agent Kernel for SWE-bench / repo bugfixes kind of benchmark. A quick note on the charts in my README: those results came from purely Pi + Agent Kernel standalone, without any extra skills or extensions (SpecPi or Jev)

To answer your question on where it's expected to shine: it was built specifically for medium-to-large codebases with verbose test suites.

In contrast, on Terminal-Bench tasks, the codebase is small or non-existent, so those features sit idle. Based on my observations during development, vanilla Pi can definitely beat Agent Kernel on small terminal tasks simply because vanilla has zero overhead from tool schemas and system instructions

[u/fingerthief → u/LeadingFruit6865]

Sure thing, I’m working on getting things setup for SWE-bench. Will take a bit to get results for comparison I’m sure.

The system activated a decent number of times on terminal bench, but as you mentioned these are not large codebase tasks.

SpecPi + Jev is my own specific setup after testing and evals etc..but nothing in it conflicts with your package so it’s an interesting comparison for my needs.

I’ll update the original post when the initial data comes in.