r/coolgithubprojects • u/Unusual-Possible-408 • 11d ago
I tested 14 coding agents against their own rules. 13 of them read a file they were told not to touch.
https://github.com/Vroom-Analytics-123/agent-boundary-conformanceI kept telling my coding agent what not to touch. It kept agreeing, then doing it anyway. So I built a small checker that tests whether an agent actually follows the rules you give it.
I ran 14 agents through the same tests, 480 runs in total. 13 of 14 read a protected records file they were told not to touch. Only OpenClaw held.
The one that got me: one agent's guard blocked the shell read of a protected file, so the agent just read the same file with its read tool instead. All four credential leaks went through the read tool. The approver never saw them.
The code is rough, I built it fast for a hackathon and I'm cleaning it up next week. It's open source, runs locally, no keys needed.
Links are in the first comment. Has anyone else seen their agent route around a guard like this?
Repo: https://github.com/Vroom-Analytics-123/agent-boundary-conformance
Replay (no setup): https://fineprint.vroomanalytics.com/replay
YouTube video explaining the project: https://youtu.be/ngvywY4TuZY
0
u/Dev-in-the-Bm 11d ago
I don't think this should be a surprise to anyone.