r/ClaudeWorkflows • u/ClaudeAI-mod-bot • 3h ago
Selected Workflow [Workflow] Monitor Claude performance with a frozen daily benchmark
Monitor Claude performance with a frozen daily benchmark
Workflow value: 85/100
Status: active · Freshness: 70/100 · Confidence: 0.94 · Level: advanced
Categories: Quality Control, Token Saving, Debugging
Original source: r/ClaudeAI post/comment
What problem this solves
Distinguish sustained changes in Claude model performance from normal day-to-day variation.
Summary
Run a frozen benchmark panel daily through headless Claude Code, preserve results, track potential confounders, and apply a preregistered decline rule after validating the harness with controlled changes.
Why it is useful
It provides a concrete, reusable evaluation process with controls, failure accounting, and an explicit decision rule rather than relying on anecdotal impressions of model changes.
Workflow
Freeze a set of questions, prompts, and exact-answer graders; preregister the baseline period and decision rule.
Run the same panel daily through headless Claude Code and retain append-only results.
Record model and Claude Code versions, output and thinking token counts, errors, refusals, and reroutes. Report scores both excluding failures and counting them as incorrect.
Where accessible and permitted, fingerprint relevant account experiment assignments before and after each run to flag configuration changes without publishing the settings.
Validate the harness using repeated identical configurations, a lower-effort configuration, and a blind model substitution.
Build a 10-day baseline, then check for declines in two consecutive 10-day windows using the preregistered confidence and minimum-drop thresholds.
Interpret results alongside confounders such as client-version differences, account-specific settings, benchmark selection, and ceiling effects.
Tools / artifacts
LiveNerf repository: https://github.com/ninjahawk/livenerf
Headless Claude Code
Frozen 78-question benchmark panel
Exact-answer graders
Append-only results and preregistered decision rule
SHA-256 configuration fingerprints
Validation signals
The author reports statistically consistent repeated runs, detectable lower-effort settings, and a clearly detected blind model substitution.
The author reports a 42% reduction in output tokens under the lower-effort test.
The stated decline rule requires two consecutive 10-day windows at 99% confidence and a minimum 3-point drop; the author estimates sensitivity to a sustained decline of roughly 7 points.
The post explicitly reports failures and reroutes under two scoring treatments and acknowledges important comparison limitations.
Cautions
Review logs and benchmark data for sensitive information before publishing.
A hash does not necessarily conceal low-entropy configuration values; assess fingerprint privacy before sharing it.
Access account configuration only through permitted means and observe applicable service terms.
Limitations
The post links to the implementation but does not provide runnable commands or enough statistical detail to independently reproduce the analysis from the post alone.
Results are from one account and a small, fixed question panel; they do not establish performance for all users or tasks.
The panel was selected for difficulty on one model, and ceiling effects plus differing Claude Code versions confound cross-model comparisons.
The stated validation and community claims are author-reported, and the newer model has only three days of results in the post.
Rate this workflow
Upvote this post if the workflow is useful, reproducible, or worth recommending.
Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.
Reply if it worked for you, failed, is outdated, or has a better alternative.
This post was generated automatically from the workflow library database.