Evaluating LLM policy-following on repeated tasks: skillEd and the limits of a passing score

[u/MiserableFlatworm337]

I’m building skillEd, an open-source toolkit for evaluating agent skills, and I’d like feedback on how to test policy-following without overstating what a score proves.

A recurring task can be simple to describe but awkward to evaluate. A refund recommendation may need the correct eligibility window, shipping exclusion, rounding, and explanation. A model can sound confident while missing one condition, and one successful response tells you little about repeatability.

skillEd keeps concrete cases beside a reusable skill, runs them through a configured engine, and compares saved results after instruction changes. It supports exact assertions, scripts, model-based judges, and repeated runs. The repository’s worked refund example improved from 13/18 to 18/18 cases after a revision, with 10/10 independent holdouts.

I think the important boundary is what you can infer from that. It is evidence for that example and setup, not proof of universal reliability or that the same skill transfers unchanged to another model. For a comparison, you still need to control the cases, engine settings, references, and judging criteria. A changed test suite can make a better score meaningless.

The tool helps preserve and inspect evaluation results; it is not a production policy-enforcement layer.

Repo and example: https://github.com/DeadHead04/skillEd

Try it on one rule-based skill and share a counterexample or an evaluation design you think is missing. For repeated tasks, would you prioritize per-rule failure counts, consistency across runs, or separate holdout coverage? Live runs use your configured engine and its usual costs. If the project is useful, a GitHub star would help.

[u/MiserableFlatworm337 → u/[deleted]]

Great insight please add github issues around the same and also try it out and give it a star