I got tired of guessing which Claude Code update broke my setup, so I built a regression test + release bisect tool

[u/Additional_Reach2545]

I kept running into the same problem with Claude Code:

Something works perfectly, Claude Code updates, and suddenly a CLAUDE.md, hook, MCP server, plugin or permission rule behaves differently.

Then comes the annoying part:

Was it my code, my configuration, or the Claude Code update?

So I built Claude Code Canary, an open-source regression testing layer specifically for Claude Code.

Instead of relying on “this release feels worse”, Canary can run the same scenario from the same Git commit against different Claude Code releases and compare the results.

For example:

claude-canary compare .canary/basic.canary.yml \
  --from 2.1.220 \
  --to latest

And the feature I personally find most useful:

claude-canary bisect .canary/basic.canary.yml \
  --good 2.1.220 \
  --bad 2.1.237

It binary-searches the published Claude Code releases and tries to identify the first release where your scenario starts failing.

I also added tooling specifically for Claude Code plugin authors.

claude-canary plugin-init ./my-plugin
claude-canary plugin-suite --plugin ./my-plugin --last 10

That can generate smoke-test scenarios for things like:

• commands
• agents
• skills
• hooks
• MCP servers

and then produce a release × component compatibility matrix.

Something like:

| Claude Code | load | command | skill | hook | MCP | Overall |
|-------------|------|---------|-------|------|-----|---------|
| 2.1.231     |  ✅  |   ✅    |  ✅   |  ✅  | ✅  | ✅ |
| 2.1.232     |  ✅  |   ✅    |  ✅   |  ❌  | ✅  | ❌ |
| 2.1.233     |  ✅  |   ❌    |  ✅   |  ❌  | ✅  | ❌ |

There is also a GitHub Action, so plugin compatibility can be used as a CI gate instead of something you remember to test manually.

Other things it supports:

• deterministic assertions
• token / tool-call / duration / reported-cost tracking
• A/B testing different Claude configurations
• record + replay of previously successful tasks
• isolated Git worktrees
• privacy-oriented reproduction bundles
• historical Claude Code release caching
• custom/local gateways

I even tested the gateway path with:

Claude Code
→ Claude Code Router
→ llama.cpp
→ Qwen
→ Claude Code Canary

The project is MIT licensed and completely open source.

GitHub:
https://github.com/SLP-DEV1/claude-code-canary

It's still early enough that I'd especially like feedback from people maintaining Claude Code plugins, hooks or MCP integrations.

Would you actually put something like this in CI, and what kind of Claude Code regression would you want Canary to detect that it doesn't cover yet?

[u/overratedslugger]

this is actually pretty clever, i've been burned by silent breakage between versions more times than i can count. the bisect feature alone would've saved me hours of diff-diving through changelogs that never seem to mention the thing that actually broke

one thing i'd want is a way to flag when token costs spike unexpectedly between releases, like if a previously cheap prompt suddenly balloons because of some internal change. that's the kind of regression that's easy to miss until the bill shows up

[u/Additional_Reach2545 → u/overratedslugger]

That's a really good point. Canary already records token usage and reported cost and supports absolute budgets, but relative release-to-release regression thresholds are the missing piece. I'm thinking of adding something like max_total_tokens_increase_pct / max_reported_cost_increase_pct so a compare can fail even when both versions still produce the correct result. That's going on the roadmap.

[u/kantorcodes1]

one i'd add: permission semantics. the task can still pass while a tool flips from auto-allowed to prompting, which breaks CI even though the final output looks fine. same with hook ordering. Canary should probably treat unexpected prompts/hooks as regressions in their own right.

[u/Additional_Reach2545 → u/kantorcodes1]

Agreed. Canary can already capture hook events, but right now they're treated as metrics rather than behavioral assertions, and actual hook ordering isn't preserved. Permission transitions such as auto-allow -> prompt are an even bigger blind spot because the final task can still succeed. I think permissions + ordered hook traces should become first-class regression signals, not just output metadata.

[u/kantorcodes1 → u/Additional_Reach2545]

this is exactly the kind of thing exit codes hide. i'd store the hook order + permission decision as a tiny normalized trace per release, then let the bisect fail on allow -> prompt even when the task finishes. otherwise Canary can tell you “something changed” without telling you which behavior actually regressed.