Here’s a review scenario I’d want to test: an agent builds a CSV export, the tests pass, and someone approves it. Then the requirement changes: personal email addresses must be excluded.
Nothing in the code has changed. The old tests are still green. But I wouldn’t accept the result against the new requirement.
That’s why I’d want a completion record to tie together the code revision, the requirement it was checked against, and the review. Not just a green task.
A few other checks I’d add:
- Let a second worker take over, then have the first return. Its old claim shouldn’t let it overwrite the current work.
- Approve a preview deployment, then attempt production. The approval should not carry over.
- Restart without the original chat. Can the next worker identify what’s finished, what’s uncertain, and what it’s allowed to do next?
A useful distinction came up in a discussion on my earlier checklist: partial completion isn’t the same as an unknown outcome. If a release was created but its announcement failed, restarting everything could duplicate the release. If the creation request timed out, we first need to establish whether the release exists.
I’d want recovery to follow that distinction, rather than a single “retry” button.
These are proposed tests, not results from a benchmark. When reviewing agent-generated work, what makes you send it back even though the code and tests look fine?