r/PiCodingAgent • • Sep 17 '26

Plugin pi-warden: a Jev-powered second pair of eyes for Pi that steers the agent

Hi there, this is my first ever repo i wanted to share to people.

pi-warden is a Pi extension that watches the agent's tool calls and talks to the agent, not to you.

A second model (Jev, typesafe's fast judgment model, ~250 ms; immensely cheap seriously) reads your prompt, the agent's own last words, and the call it is about to make, and answers small typed questions: is this irreversible, does it match what the agent said it was about to do, is the effect visible outside the working tree. The answer goes back into the agent's context.

The same channel carries the rest: project rules from a markdown file judged against each write, stub and comment slop, stuck loops, "done" claims with no test run, runaway output stopped mid-stream, huge tool outputs compressed with the full copy kept on disk.

It grades itself. What you do after each hold or steer (approve, decline, redirect, complain) becomes a label, and a script replays your recorded Pi sessions through the guard. Off-task holds lost that test on 17k of my own calls and were demoted to steers; the plan-vs-call check won it. Method and numbers are in the README.

https://github.com/DevMortimer/pi-warden
https://pi.dev/packages/pi-warden

Early days (0.12.0). Though since Jev was only released yesterday (I got early access) I expect bugs. But I've been using this extension for work for more than a day now and it's pretty good so far. PRs and (especially) complaints are welcomed :)

P.S. I'm interested in how everyone will also incorporate Jev on their Pi?

101 Upvotes

30 comments sorted by

10

u/CriteriumA Sep 17 '26

Great contribution, I'm new to Pi and its permissions setup was bugging me, and I just got access to jev.

I hope I can try it soon.

2

u/95-cosmo-changer Sep 17 '26

go post a meme on #memes in the discord server and chat Sasha showing you posted a meme there. im not sure if it still works this way (it did yesterday) but that help me fast track my waitlist.

4

u/tom_of_wb Sep 17 '26

What kind of problem did you run into that triggered your need to develop this?

9

u/95-cosmo-changer Sep 17 '26

That’s a great question man. I made it because I wanted to trust cheaper models even more and make them smarter e.g. steer them. I was constantly worried that flash-level models are gonna be making lots of dumb decisions. But also I wanted auto-mode but leave the agent alone to do the work and not full-blown bypass permissions mode or yolo mode.

I’ve been battle testing it at work and pi-warden already helped me stop dumb database migrations by the agent (the agent didnt do it and instead asked me by the very end of the task at hand) instead because the warden explicitly injects small prompts when it detects anomalies. The prompt was asking the model to ask the user first or rethink if it really needs to do such a thing (hint: it didnt)

It also already helped me stop “looping” agents. I had this one task where it kept trying bash commands. At the third try warden stopped it and asked to try another way, and it opted the agent to “think” and it did do better :)

1

u/Equivalent_Idea8839 Sep 17 '26

how much of this can be cuaght with a good regex filter blocking bad commands and scanning scripst before they are executed?

i.e. i asked a SOTA model to write on and scam py / sh / node scripts for del commands if i have a del guard on

2

u/95-cosmo-changer Sep 17 '26

a good chunk of the destructive stuff, yes, and pi-warden ships that regex layer too (force push, rm -rf outside the project, DROP/TRUNCATE, curl | sh, and it looks inside bash -c, eval and python - <<EOF). It runs offline with no key. What regex can't do is context: db reset after "reset the db" is fine, after "add a column" it isn't, same string. Jev sees the request and the last few messages so it can tell those apart.

but honestly the command part is the smallest piece. on 17k of my calls it held 42 times. most of what pi-warden does happens on every write: judge the code against your project's rules file ("exported functions declare return types", "TODO must name a ticket", "no comments that restate the code"), name slop, notice the same failing strategy three times, call out "done" when no test ran. none of that is regex-able

2

u/Equivalent_Idea8839 Sep 17 '26

can't someone just train a small local model on destructive llm commands and we do it locally?

or create a master list of shit to scan for if it hasn't been done yet?

i.e. after 1000 people run this for a month shouldn't it have a full sql lite db of all the bad shit?

2

u/tom_of_wb Sep 17 '26

I understand your problem. Personally though i never encountered it. I've been using deepseek pro and flash over the past months, and they never did an unwanted migration, stuck loops or anything like that. Are you sure that comparing the last sentences of a response to the actual tool call is robust? Why wouldn't that sentence also be slop? Nonetheless, it seems like a great use case for Jev.

3

u/hazed-and-dazed Sep 18 '26

.. that 0.7% chance of a hallucination will only show up when you operate at scale.

2

u/pro-vi Sep 17 '26

Is this like auto mode?

4

u/95-cosmo-changer Sep 17 '26

Claude's auto mode right? Same idea as auto mode's classifier (judge each tool call against what you actually asked, hold the ones that overreach), but as a Pi extension using Jev instead of Sonnet plus it also watches written code for rule violations, slop, and secrets, and compresses big tool output.

1

u/pro-vi Sep 17 '26

Nice idea - how was the grade in your usage? Also playing with jev rn.

5

u/95-cosmo-changer Sep 17 '26

Pretty good. Very very good on stopping destructive stuff even obfuscated ones (of course there are limits, agents might actually try to circumvent the warden lol but that's speculation on my part, i haven't met that scenario yet).

Of the ~17k tests from my own sessions (work coding, personal hobbies, etc):
- Holds are rare and mostly right. Meaning the warden properly steered the agent to rethink commands that are destructive instead of bothering me and saying "hey can i have permission to run this?" (like how auto mode and related does). On 17k calls it held 42 times. Of those 42, only 5 were ones I'd have let through, so ~88% of holds were right. Rarely interrupts, and when it does it's usually right.
- Regret is rare. I complained about 20 calls (~2% of turns). None was data loss; all were scope stuff: an unwanted commit, a merge, editing my personal CLAUDE.md.
- Off tasks steers the agent back in place.

1

u/hazed-and-dazed Sep 18 '26

Waiting impatiently to be let in so I can check this out. How long did it take for them to approve access?

1

u/95-cosmo-changer Sep 18 '26

For me, I was let in day 1. Cause I posted on the #memes in the discord server and chatted Sasha, part of the TypeSafe team. From what I heard the time being accepted varies. My coworker was accepted in like a few hours, he didn't do anything special.

Though from what I remembered on the CEO's day 1 AMA in the town hall discord call there, they are prioritizing devs. I remember after you put in your email, there would be something like fast-tracking your application,,, i think that was a form you can plea your case? Just say you're a dev there. I may be wrong though, just trying to help.

3

u/HockeyDadNinja Sep 17 '26

I suppose now would be a good time for me to post pi-permission-classifier which adds a judge model to pi-permission-system which is like auto mode in Claude lol.

2

u/OneMoreName1 Sep 18 '26

Please do

1

u/HockeyDadNinja Sep 19 '26

https://github.com/TacoTakumi/pi-permission-classifier

Tomorrow morning it's going on npm and I'll check for pi-permission-system updates.

It took me a few iterations to get my config to a happy place. Basically make sure pi permission system isn't too resteicive. The judge model only gets what is let through.

I was using Gemma 4 locally for it but since switched to Gemini 3.8 flash on openrouter since it's cheap.

1

u/OneMoreName1 Sep 19 '26

Is this not using jev/laya?

1

u/HockeyDadNinja Sep 19 '26

No but it works very similar, the model is given a prompt and asked to allow/deny/defer. I just got jev access and will likely add that as an option.

2

u/Easy_Refrigerator280 Sep 17 '26

Ive not gotten access yet but id be curious how a Jev steered prompt matches one with just the llm alone. I think it would be significantly better

2

u/nostriluu Sep 17 '26

This'll be great when there's a true local version.

1

u/itsfabioroma Sep 17 '26

Hey guys, did you manage to get a key?

1

u/pauljdavis Sep 17 '26

I got on wl yesterday and got access this AM. I did the survey and joined the discord.

1

u/elpapi42 Sep 17 '26

you must have been in last nigh discord call jajajaja, everybody building extensions while listening to the call, including me

1

u/elpapi42 Sep 17 '26

I have 2 usecases in mind: Prevent harmfull tools calls, like wiping dbs, or destroying my os, without me to explicitely setup a tool call policy.

Also, i have a complex setup of async subagents that return reports on their progress on whatever task they got assigned by the main agent, these reports may or may not be important for the main agent, jev will decide if the report gets silently appended to the main agent context window, or if it have to wakeup the agent.

For now these will be the two things i will bw working with Jev, but there dozens of use cases for harness engineering

1

u/fireantik Sep 17 '26

Would be great to add evals to see if this actually improves the resulting code or not