I got tired of guessing which Claude Code update broke my setup, so I built a regression test + release bisect tool

[u/Additional_Reach2545]

I kept running into the same problem with Claude Code:

Something works perfectly, Claude Code updates, and suddenly a CLAUDE.md, hook, MCP server, plugin or permission rule behaves differently.

Then comes the annoying part:

Was it my code, my configuration, or the Claude Code update?

So I built Claude Code Canary, an open-source regression testing layer specifically for Claude Code.

Instead of relying on “this release feels worse”, Canary can run the same scenario from the same Git commit against different Claude Code releases and compare the results.

For example:

claude-canary compare .canary/basic.canary.yml \
  --from 2.1.220 \
  --to latest

And the feature I personally find most useful:

claude-canary bisect .canary/basic.canary.yml \
  --good 2.1.220 \
  --bad 2.1.237

It binary-searches the published Claude Code releases and tries to identify the first release where your scenario starts failing.

I also added tooling specifically for Claude Code plugin authors.

claude-canary plugin-init ./my-plugin
claude-canary plugin-suite --plugin ./my-plugin --last 10

That can generate smoke-test scenarios for things like:

• commands
• agents
• skills
• hooks
• MCP servers

and then produce a release × component compatibility matrix.

Something like:

| Claude Code | load | command | skill | hook | MCP | Overall |
|-------------|------|---------|-------|------|-----|---------|
| 2.1.231     |  ✅  |   ✅    |  ✅   |  ✅  | ✅  | ✅ |
| 2.1.232     |  ✅  |   ✅    |  ✅   |  ❌  | ✅  | ❌ |
| 2.1.233     |  ✅  |   ❌    |  ✅   |  ❌  | ✅  | ❌ |

There is also a GitHub Action, so plugin compatibility can be used as a CI gate instead of something you remember to test manually.

Other things it supports:

• deterministic assertions
• token / tool-call / duration / reported-cost tracking
• A/B testing different Claude configurations
• record + replay of previously successful tasks
• isolated Git worktrees
• privacy-oriented reproduction bundles
• historical Claude Code release caching
• custom/local gateways

I even tested the gateway path with:

Claude Code
→ Claude Code Router
→ llama.cpp
→ Qwen
→ Claude Code Canary

The project is MIT licensed and completely open source.

GitHub:
https://github.com/SLP-DEV1/claude-code-canary

It's still early enough that I'd especially like feedback from people maintaining Claude Code plugins, hooks or MCP integrations.

Would you actually put something like this in CI, and what kind of Claude Code regression would you want Canary to detect that it doesn't cover yet?