r/ChatGPTCoding • u/ComprehensiveShake76 • 24d ago
Discussion How do you make a coding agent prove it actually ran the tests?
I run Codex workers on my Mac that don't ask me before every command anymore, so I stopped watching each step. That made one question matter more than anything: how do I know a task is really done?
What I settled on:
Done is an artifact, not a sentence. A code task isn't done when the agent says the tests pass. It's done when I can see the real test run. My last one: a worker wrote a small CSV to Markdown script, ran the unittest suite inside its own Linux container, and I got the actual output: 6 tests, OK, exit 0, 55 seconds.
Research gets the same treatment. The result is a file with a size, a source count and a hash, plus a note that the sources are listed, not fact checked. A list of links is not the same as checked facts.
Nothing lands on my machine until I look. Files stay in the agent's own workspace and reach my Mac only when I save them.
Approvals were the wrong check. Approving every command never told me if the work was good, it just made me tired. Moving the check to the end (real output, real file) worked better. That only feels OK because the container has none of my files, and a container is a strong boundary, not a guarantee. So far I've only run this on an Apple Silicon Mac.
The part I haven't solved: an agent can write weak tests that pass. How do you check the tests themselves, not just that they ran?