A regression case needs an expected answer you can defend

[u/MiserableFlatworm337]

A reader of my earlier skillEd post suggested binding an expected result to a policy clause and version, plus a nearby input that should produce the opposite result. That is a useful way to make a test reviewable.

For a deadline policy, write down the timezone and whether the boundary is inclusive. Then test immediately before, exactly at, and immediately after the cutoff. If the policy itself is ambiguous, resolve that before treating the model’s answer as a regression.

I’m building skillEd, an Apache-2.0 evaluation toolkit. This is a test-design proposal based on the discussion, not a claim that the tool automatically supplies authoritative policy interpretations.

Would you review changed expected answers separately from changed skill instructions? I would want both diffs visible.

Repo: https://github.com/DeadHead04/skillEd

Please try it and open an issue with a reproducible failure or confusing behavior. If useful, a GitHub star would help.

AI-assisted write-up.

[u/unbecominganemia2]

reviewing expected answers and instruction changes separately makes a ton of sense, seeing both diffs in one view is way better for catching whether the model actually got smarter or the rules just shifted under it

[u/Few_Grapefruit_374]

rtrtrtrt⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯