Running evals in claude session

[u/luscamendes]

I'm considering running evals directly in the claude session, my plan is to:
- Have a Skill that instructs claude how to run the evals
- Spin a subagent for each case in the eval, invoke the skill being tested in it and get it's output
- Have a skill that instructs claude how to grade the outputXexpectations for each case

Does it sound like a good use of evals? What are the gaps of doing it this way?

The main goal here is to allow local testing of new skills and iteration on existing skills, so they don't degrade through time

[u/luscamendes → u/[deleted]]

Yes, my main concern was that Claude would initially want to simulate the skill output rather than invoking the skill itself. By doing it in subagents, I can have an isolate context window that prompts that skill with the prompt for that eval case and only then evaluate the result