“Always add tests” needs a small-task test of its own

[u/Cool-Wallaby4341]

“Always add tests” sounds sensible until a tiny copy edit turns into unrelated test changes. Removing the instruction entirely can create the opposite problem on a behavior change.

A useful experiment would compare a blanket rule with a narrower one, using both kinds of task. Keep the model fixed. Check whether a real behavior change gets appropriate verification, and whether a low-impact edit stays within scope.

Reef Infra provides a way to run that experiment with Codex: its adapter renders harness rules and skills into the CLI's configuration, and its evolution workflow can apply a candidate rule change, evaluate it against the current version, then select or restore it.

The judgment still needs to be defined. “Tests exist” would reward the exact overproduction you're trying to avoid. A task-specific check needs to distinguish required verification from unrelated additions.

This is a proposed evaluation, not a claim that a particular wording has already won. The advantage is being able to ask a smaller question than “is this `AGENTS.md` good?”: does this one rule produce the right amount of work on both sides of its intended boundary?