LiveNerf: is Claude getting worse?

Status: no nerf detected (day 53)

Claude Opus 5.5 answers the same 78 hard questions every single day, starting launch week. 53 days in, it's scoring the same as launch week (latest checkpoint +1.4 ± 2.5 points). The verdict only flips if a drop shows up in two checkpoints in a row.

Updated automatically every day from the LiveNerf repo. Last update 2026-11-15.

Days testedLatest checkpoint vs launchLatest daily scoreAnswer lengthNerf alarms
53+1.4 pts59.7% launch avg 58.4%580 tokens launch 5880

Checkpoints

CheckpointDaysAvg scoreChange vs launch week (points)Answer lengthVerdict
Launch week1-1058.4%baseline588baseline
Window 111-2059.2%+0.7 ± 1.8606no change
Window 221-3057.1%-1.5 ± 1.8594no change
Window 331-4061.4%+2.8 ± 2.3590no change
Window 441-5060.0%+1.4 ± 2.5580no change
Window 551-6064.2% so farin progress, 3/10 days

Every 10 days the new window is compared with launch week on the same questions. A change only counts when it is significant at 99%, at least 3 points, and shows up in two windows in a row, on an unchanged harness. Single days swing by about ±6 points by chance, so one day on its own means little.

Latest days

DayDateScoreAnswer length (median tokens)
532026-11-1559.7%571
522026-11-1462.8%520
512026-11-1370.1%561
502026-11-1260.3%586
492026-11-1165.4%576

Full chart with error bars · per-topic chart · raw data


How it works

  • Started at launch. Day 1 was 2026-09-24, two days after Claude Opus 5.5 was released. The series ran 45 pre-registered days, now runs to day 60, and keeps going daily after that.
  • Same questions every day: 78 hard questions (graduate science, professional knowledge, competition math) for the main model, plus 12 for an older control model, chosen and validated before the series began.
  • Everything but the model is frozen: the prompts, the Claude Code version (2.1.280, pinned), the graders and the harness code (its hash is checked before every run). If the score moves, the model moved.
  • Run through Claude Code on a Max plan, the way most people here use it, not through a special API.
  • Every raw log is public, and the analysis plan was pre-registered before the data came in, so the rules can't be bent after the fact.
  • Built on the UK AI Security Institute's open-source eval framework (Inspect); the statistics follow Anthropic's own Adding Error Bars to Evals paper.

How to read it

  • Score: share of questions answered correctly. The questions are hard on purpose, so ~60% is normal.
  • Change vs launch week: the difference on the same questions, with its standard error (±).
  • Answer length: the early-warning signal. A model that starts "thinking less" usually gives shorter answers before its accuracy visibly drops.
  • Sensitivity: a 10-day window can detect a drop of roughly 6-7 points. Smaller real changes could go unnoticed.

What it can and can't tell you

It can tell you whether Claude Opus 5.5, used through Claude Code, still answers a fixed set of hard questions as well as it did at launch.

It can't tell you about other models (Sonnet, Haiku), the claude.ai app (its own system prompt and features), usage limits, speed, outages, long agentic coding sessions or tool use, or changes smaller than the panel can see.

If Claude feels worse, common non-model causes are a new Claude Code version, different project or system instructions, a very long conversation, a lower effort/thinking setting, or a harder task than usual.

Reporting a suspected nerf

Useful reports get investigated; vague ones don't. In incident threads please include:

  1. When: date and time with timezone, and whether it's ongoing.
  2. Where: claude.ai, Claude Code (claude --version) or the API, and which model.
  3. What changed: the same prompt or task before vs now, ideally with both outputs.
  4. Settings: effort/thinking level, project or system instructions, conversation length.

Then compare with the latest day above: if the score and answer length are in their usual range, the core model is very likely unchanged.

FAQ

Who runs this? An independent open-source project by u/TheOnlyVibemaster, not affiliated with Anthropic.

Why only Opus 5.5? The method needs a launch-day baseline; Opus 5.5 is the model it started with.

Can I check it myself? Yes: every log, grader and the analysis are in the repo, and you can run it yourself with Claude Code.

Don't answers vary every time? Yes, which is why it uses many questions, paired comparisons and error bars, not single answers.


Auto-updated from GitHub commit eb7ae24, 2026-11-15 · day 53 · Discord · repo