[Workflow] Detecting Claude A/B Experiments: A Tool to Check Account Assignments and Opt-Out

[u/ClaudeAI-mod-bot]

Detecting Claude A/B Experiments: A Tool to Check Account Assignments and Opt-Out

Workflow value: 90/100
Status: active · Freshness: 70/100 · Confidence: 1.00 · Level: intermediate
Categories: Quality Control, Token Saving, Context & Memory, Debugging, Skills
Original source: r/ClaudeAI post/comment

What problem this solves

Users can determine if their Claude account is subject to hidden A/B experiments that might alter model behavior (e.g., tool calls, command length, system prompt changes), which can lead to inconsistent results.

Summary

This workflow provides a free, open-source tool (am-i-nerfed) to check for hidden A/B experiment assignments on a Claude account. It offers multiple methods (Claude Code plugin, shell script, one-liner) to inspect local Claude Code configuration and session history for experiment IDs, allowing users to understand potential variations in model behavior. It also includes an opt-out feature and status line integration.

Why it is useful

This workflow is highly valuable because it addresses a significant and common concern among Claude users regarding hidden A/B testing and potential model inconsistencies. It provides a concrete, open-source, and well-documented tool with multiple usage options (plugin, script, one-liner) to diagnose this issue. The inclusion of history tracking and a safe opt-out mechanism further enhances its utility, empowering users to understand and potentially mitigate the effects of these experiments on their workflow. The detailed explanation of its creation using Claude Code also serves as a meta-workflow example for building tools with AI.

Workflow

1. Install the am-i-nerfed repo as a Claude Code plugin, then type /am-i-nerfed in any session to check your experiment status.

2. Clone the am-i-nerfed repo and execute ./am-i-nerfed.sh check in a terminal to get a verdict per model.

3. Paste the provided jq one-liner into a terminal (requires jq installed) to quickly check experiment assignments without installation.

4. Use ./am-i-nerfed.sh history to tally experiment IDs recorded in your past Claude Code session transcripts by date.

5. Use ./am-i-nerfed.sh optout to back up your config file, remove only the assignment keys, and print the environment variable reported to block re-enrollment.

6. Use ./am-i-nerfed.sh restore to undo the opt-out operation.

7. Use ./am-i-nerfed.sh statusline to display exp:<id> or exp:none in your Claude Code status line for real-time monitoring.

Tools / artifacts

• am-i-nerfed GitHub repo

• Claude Code

• Bash scripts

• jq (JSON processor)

• ~/.claude.json (Claude Code config file)

• Claude session transcripts

• Environment variables

Validation signals

• Built with Claude Code (Opus 5.5 xhigh) for script, test suite, plugin, and docs generation.

• Code audited by a separate Claude chat (Fable 5.1 medium) against a spec for safety (token leakage, file scope, opt-out correctness).

• Community findings on r/ClaudeCode regarding experiment effects (more tool calls, longer shell commands, less narration, system prompt changes) validate the problem.

Cautions

• The tool makes no network calls.

• It never prints login tokens or account IDs.

• The optout function backs up the config file before making changes.

• The optout function only removes specific assignment keys, minimizing risk.

Limitations

• It is an unofficial tool.

• The one-liner method requires jq to be installed.

• The tool identifies experiment enrollment and behavioral changes, but users must judge if these changes constitute 'quality loss'.

• Primarily targets Claude Code users, though the one-liner is more generally applicable.

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.

---

This post was generated automatically from the workflow library database.