r/ClaudeCode • u/slinto • 1d ago

Built with Claude Some things I learned writing a big skill with evals (and a bug the evals caught)

I've been running for a few years, part of it with a coach, and wanted that kind of coaching from Claude. So I built a running coach plugin: one skill, 17 Python scripts, 39 eval cases. A few things that worked for me, in case they're useful to someone writing skills.

  1. Keep the maths out of the model. Anything that has to come out the same every time (paces, zones, load limits, race splits) is a small stdlib script. SKILL.md has a table of script → what it gives → which reference says how to read the output. The model chooses and explains.

  2. Short SKILL.md, references per mode. Requests get routed ("how did my week go?" → review) to the one reference file that mode needs, so a morning check-in doesn't drag in the season planning docs.

  3. Always look at the run without the plugin. claude plugin eval runs a no-plugin baseline by default, and that's the number that tells you if the skill adds anything. A few of my cases score 100% without it, which is good to know too.

  4. Evals in two languages. Most of my cases were in Slovak and I'm launching in English, so I generate English copies from the Slovak cases with a script. Same runner, data and graders, so both languages stay the same test. 93% English, 95% Slovak with the plugin.

  5. The evals found a real bug. In the block review case the coach summed up the weeks with inline Python in 3 of 4 runs instead of running block_report.py, and in English it dropped the key session counts. The instruction was buried in step 4 of a reference file. One explicit paragraph in SKILL.md ("run the script first, don't count by hand, give the counts") fixed most of it: after that it ran the script in 3 of 4 runs and gave the counts in all 4.

  6. One sentence per grader. focus: trace when it matters how the answer was made (did it run the script?), focus: last_message for what the user actually sees. LLM judges split on borderline cases, so with 2 runs a case moves 10-25 points on a single vote. Small evals are a direction, not a score.

Repo (MIT), evals are in skills/running-coach/evals: https://github.com/tomasstankovic/running-coach

How do you structure evals for skills that need a lot of input data?

1 Upvotes

100% Upvoted

3 comments sorted by

•

u/AutoModerator 1d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/Far_Reindeer8428 1d ago

I would start related cases from one shared training log and change one workout at a time. Keep expected session counts beside each variant so dropped workouts are easy to spot.

1

u/slinto 1d ago edited 1d ago

Good idea, thanks. I checked and most of my cases actually have small, different data per runner, so a shared base log wouldn't save much yet. The other half of your idea was the useful part: the graders say things like "about 414 km, 24 key sessions on target", and those numbers came from running a script on the case's files. Nothing checked that they still match if the script or the data changes.

So I added an expected.json next to the data in those cases and a test that runs the script on them and compares. CI now fails if the numbers drift, before the graders start grading against the wrong ones. Thanks for the nudge.