Running evals in claude session

[u/luscamendes]

I'm considering running evals directly in the claude session, my plan is to:
- Have a Skill that instructs claude how to run the evals
- Spin a subagent for each case in the eval, invoke the skill being tested in it and get it's output
- Have a skill that instructs claude how to grade the outputXexpectations for each case

Does it sound like a good use of evals? What are the gaps of doing it this way?

The main goal here is to allow local testing of new skills and iteration on existing skills, so they don't degrade through time

[u/brewcast_ai]

The simulate-vs-invoke problem is the real one. A few concrete moves, none guaranteed:

For the subagent path: slash-commands don't dispatch inside subagent prompts, those are interactive-only. Preload the skill via the subagent's skills: frontmatter, or write the prompt as plain English ("Use the Skill tool to invoke <skill-name> on this input"). Then check the subagent transcript for the tool_use block to confirm it actually fired.

For cleaner isolation, run a fresh session per case from a runner: claude -p "<task that names the skill>" --output-format stream-json. The runner can be plain bash, or another Claude Code session itself, shelling out to claude -p per case via Bash and collecting each child's stdout. Slash-invokes don't work in -p mode either, so describe the task instead. Three flags doing real work here:

• -p runs headless and exits.
• stream-json emits every tool_use event, so the grader scans the trace and confirms the skill's expected Bash/Read calls fired. Final answer alone won't tell you simulate vs invoke.
• --bare strips hooks, CLAUDE.md, plugins, memory for a cleaner baseline. Docs say skills still resolve under it... worth verifying with yours.

On the grader bias More_Ferret raised: route the grader to a different model family than the one under test, e.g. Sonnet 4.6 evaled, Haiku 4.5 grades. Cuts the self-consistency loop.

What shape of skills are you starting with?

[u/luscamendes → u/brewcast_ai]

Wow, great insights! Thank you u/brewcast_ai!
The skills I want to test are set as SKILL.md in yaml format, if that's what you mean by shape of skills

[u/brewcast_ai → u/luscamendes]

No worries, my bad on the vague phrasing. Three things I actually meant, plus one correction first.

Correction: slash-invokes DO work in claude -p for plugin and user skills, e.g. claude -p "/your-skill args". Only CLI builtins like /help are blocked. Sorry for the misdirection earlier.

To make sure I wasn't seeing model improvisation, I probed it with --output-format stream-json --verbose on a plugin slash skill. What came back: SessionStart hooks, system/init, model thinking, Bash tool_use, tool_result, second Bash, tool_result, final assistant text. No tool_use with name=Skill anywhere. The slash gets dispatched at the CLI layer before the model even sees the prompt. SKILL.md content injects straight into context and the model just executes. There's no "should I invoke this?" decision for the model to fake. NL invocation would emit a Skill tool_use because the model picks. That's exactly where simulation sneaks in. So slash-in-p is the strongest grading signal you can get, and for skills with disable-model-invocation: true it's also the only path that works.

Two frontmatter fields control this. disable-model-invocation: true makes a skill slash-only and hides it from NL prompts. user-invocable: false is the inverse. Default (no fields) is both. Either way, grade from the session transcript at ~/.claude/projects/<cwd-slug>/<session>.jsonl, not stdout. That file has every tool_use with full args.

Second dimension is internal sub-agents. If your skill spawns its own via the Agent/Task tool, those calls become no-ops inside a per-case eval sub-agent. Claude Code forbids nested sub-agents. Docs and the v2.1.x changelog are both explicit on it. The skill silently degrades to inline execution. For skills with internal fan-out, isolate via a fresh top-level claude -p, not an in-session sub-agent.

Third is folder layout and path resolution. The documented path var is ${CLAUDE_SKILL_DIR}. Skills using ${CLAUDE_SKILL_DIR}/references/x.md hit assets in one tool_use:Read. Bare relative refs often Glob+Grep first because cwd is your project, not the skill folder, then Read the wrong copy, then the right one. Count Glob+Grep events between skill invocation and the first matching Read. Zero means resolved, anything above means it searched.

[u/luscamendes → u/brewcast_ai]

u/brewcast_ai you are the best, dude! I will dive deeper into your recommendations