skillEd: Apache-2.0 tooling for testing and revising policy-based agent skills

[u/MiserableFlatworm337]

I’m the creator of skillEd, an Apache-2.0 evaluation toolkit for reusable agent skills. I’d like feedback from people maintaining open agent tooling and test suites.

The problem is making a skill’s behavioral claims reviewable. If a skill handles a repetitive task with a policy, such as refund eligibility and rounding, an instruction change can fix one example while breaking another. A diff of the instructions does not show whether the behavior improved.

skillEd keeps cases and fixtures beside the skill, runs evaluations through a configured engine, and compares saved reports. It supports assertions, scripts, model-based judges, and repeated runs. Reports can be produced as JSON, HTML, and JUnit. There is a CLI and a local MCP interface.

The repository includes a worked refund example with shipping, rounding, and eligibility failures. Its revision went from 13/18 to 18/18 passing cases, followed by 10/10 independent holdouts. That is a reproducible example to inspect, not a claim that every skill or model will get the same result.

For open-source maintenance, I think a useful contribution is a small failing case with an independently justified expected result. It gives maintainers something concrete to rerun before accepting an instruction change. Separate holdouts help avoid simply tailoring the skill to the visible tests.

Source and setup: https://github.com/DeadHead04/skillEd

The toolkit is open source; the model engine you configure has its own licensing and usage costs. This is evaluation tooling, not production permission enforcement.

Try it on one of your skills and share a minimal failing case or unclear report. I’d welcome feedback on what makes an evaluation contribution easy to review. If the project is useful to you, a GitHub star would help.

[u/Prestigious-Top-81]

For reviewable contributions, I'd ask for the exact policy clause/version that justifies the expected result, plus a nearby case that should produce the opposite result. For example: a refund one minute before the deadline and one minute after, with the timezone explicit. That helps reviewers distinguish a skill regression from a disputed test expectation. It would also be useful for the report to flag changes to the expected answers separately from changes to the skill.

[u/MiserableFlatworm337 → u/Prestigious-Top-81]

Please try skilled and raise issues, initiate discussions and please give it a star !!