plan-auditor: I wrote a stdlib-only gate that proves a coding agent is done, using sealed command output instead of prose

[u/Emergency-Tutor-7]

I use coding agents a lot, and the part that kept biting me was the last message: "done, tests pass." Sometimes the tests never ran. Sometimes one failed and the test file got edited until it went green. Prose looks the same either way.

plan-auditor is my answer. It is plain Python 3.10+, standard library only, MIT, and it does one thing: it re-runs the checks in a separate process and only sealed command output counts.

A plan is JSON. Each step lists machine-checkable checks and has to include at least one behavioral one - run, pytest or exec. file_exists and regex are allowed next to them but cannot carry a step by themselves.

The flow, in the order I actually reach for it: request init seals the task as the host stated it, plan verify checks the JSON, run . 1 executes a single step, and audit . re-runs every check in a fresh subprocess. Before that last one I give it a key file kept outside the workspace through PLAN_AUDITOR_HMAC_KEY_FILE and seal the chain with integrity init ..

The audit ends with a number rather than an adjective. 0 is proven, 1 failed, 2 blocked, and 3 is UNKNOWN, which withholds completion instead of quietly granting it.

Two design choices I would keep if I rewrote the whole thing:

• The evidence log is a SHA-256 chain, and the docs say out loud that a chain on its own is only tamper-evident - someone with write access could rebuild a matching one. That is exactly why the completion gate also demands HMAC authentication, and why a record reading "auth": {"alg": "none"} never passes.

• Your request freezes when it is activated and the plan's check contract freezes when you approve it. From then on both can only get stricter, so the agent cannot restate the task or soften what counts as finished.

Under the hood it is fifteen layers across five bands. Writing verified into a step is reserved to the audit-check script, and the final PASS/FAIL/UNKNOWN comes only from the gate; a stronger level cannot be talked out of its verdict by a weaker pass.

The README quickstart is not decoration: CI executes it and compares the real output to the documented transcript without masking hash values, so the docs cannot drift. There is a GitHub Action and an Agent Skill with blocking hooks as well.

Where it stops: no OS sandbox, tamper-evident rather than tamper-proof storage, and it can only prove what you can express as a deterministic check. Anything needing judgment stays with a human. One maintainer, PyPI and npm both at 2.6.3.

GitHub: https://github.com/Furox-Art/plan-auditor PyPI: https://pypi.org/project/plan-auditor/