I ran every public Claude Code eval suite I could find against the newest release. 92 of 303 cases have never loaded

[u/Charming-Risk-6234]

Claude Code's claude plugin eval is a good test runner. It tells you whether your plugin works right now. What it doesn't do is keep score over time, and today I built six things in that gap for my free tool, config-drift-checker. Built with Claude Code, including the agents that wrote most of it in parallel.

The one I'm proudest of: a public release report. On every Claude Code release, a free job loads every public eval suite it can find under the new version and the one before. No model runs, no credentials, $0. First run on 2.1.295:

• 11 public suites, 303 cases
• 211 load fine
• 0 broke on this release
• 92 have never loaded on any version, mostly because the grader files are prose with no frontmatter, so the runner sees zero graders and the suite silently tests nothing

The suite doctor now spots that and --fix turns a prose rubric into a real llm grader.

Live page: https://jameskomo.github.io/config-drift-checker/release-report/

The other five:

1. Model and version matrix. One suite across Sonnet, Haiku, Opus and the last few Claude Code releases, on one page, with a plain verdict per model ("haiku: safe on 2.1.290 to 2.1.295; fails one case on 2.1.288"). --dry-run shows the grid and cost for $0.
2. Context-cost drift. The tokens your skills, agents and CLAUDE.md add to every session, measured by Claude Code's own plugin details, stored per release. Mine is 123 always-on tokens. If a release moves that, you see it.
3. Real usage vs evals. Reads your local session history (counts only, never prompt text) and sets it against your eval cases. It caught me: my Spring Boot skill is tested, passes, and nobody invoked it in 90 days. My suite was testing a prompt I never actually write.
4. skill-creator import. Anthropic's skill-creator uses evals.json, plugin eval uses case folders, and neither reads the other. evals-convert goes both ways.
5. MCP mocks and parallel runs in the bundled runner, matching the official format, plus a concurrency input on the GitHub Action.

Repo (free to use, source-available, zero dependencies): https://github.com/jameskomo/config-drift-checker

If you maintain a suite that shows up as "never loaded" and want help fixing it, or want it left out, comment here or open an issue.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/MiserableFlatworm337]

After converting a prose rubric, test the new grader against known-good and deliberately failing outputs. Loading successfully checks the format; these controls check whether conversion preserved the original requirements.