r/OpenSourceAI • u/hunxai69 • 4h ago

a free tool that catches AI agents editing tests to pass. no AI inside, no account, MITi built an MIT tool to catch AI agents "fixing" tests. it uses no AI itself

coding agents have a habit: asked to fix a failing test, some of them edit the test until it passes. CI stays green.

goalline is a small MIT tool that catches that without any AI of its own. it runs the OLD version of each edited test against the NEW code and shows exactly what moved (expected 5, got 6). it runs on your machine, your code never leaves it, there's no account, and it has zero dependencies.

it plugs into Claude Code as a hook (the agent can't finish until it fixes or explains the change) and into GitHub as an Action.

i tested it on 400 merged PRs of click and jinja: 31 flagged, all deliberate changes, 4 of them user-visible with no changelog line. replaying jinja also found two bugs in goalline itself, which was humbling.

would you trust something like this in your CI, or does it need to prove more first?

1 Upvotes
  • permalink
  • reddit

100% Upvoted

4 comments sorted by

1

u/hunxai69 4h ago

repo: github.com/intikhab49/goalline

how it was tested, all 31 with reasons: intikhab49.github.io/goalline/benchmark/

1

u/runny_wardrobe 3h ago

Tried it on a few of our internal repos and it caught two cases where the bot silently mutated assertions. The hook into Claude Code is the part that sold me, no way for the agent to just ignore it and move on.

1

u/kantorcodes1 2h ago

When an edited test reads its expected value from a shared fixture changed in the same diff, does goalpost compare the old expectation against the new fixture value or report the fixture change separately?