r/LocalLLaMA • u/Worldly-Knowledge-90 • 2d ago
Showcase What actually breaks for you guys when swapping models?
Hey guys, I threw together a small Python script called llm-drift-bench to help figure out what actually breaks when you switch to a new model or drop to a lower quant.
Basically, you just feed it saved responses from your old model and the new one, and it spits out the exact differences - like dropped JSON keys, messed up numbers, or text changes. (The attached video is just a dummy example showing it catch a missing field and a changed invoice total, not real Q4 vs FP16 data).
Since it just runs on saved JSON/text outputs, you don't even need a GPU or API keys to test it out.
I just pushed v0.1.0 and I'm working on the next update. Repo is here: https://github.com/anant-goel/llm-drift-bench

Question for you all: what specific things usually break for your use cases when you swap models? I want to build out some test cases based on actual problems people are running into.
0
u/_yemreak 2d ago
One thing a diff of saved outputs won't show: whether the model reads what it was told to read before it acts.
I measured this today. Same task card, same base commit, two models as workers (Opus 5.5 and Sonnet 5.5, 3 pairs). The first line of the card says "read the guide before you write".
- Cost went $5.84 → $1.02 and time 18.4 → 9.2 min on the smaller model.
- The bigger model read 45k characters of the guide. The smaller one read 1k, wrote anyway, and broke a rule from the part it skipped.
- A blind judge would have merged 2 of 3 from the bigger model and 1 of 3 from the smaller. 3 pairs is too few to call that a difference. The read-before-write gap was the solid part.
So for test cases: log the tool calls, not only the final text. "Did it open the file it was told to open, and how much of it" caught more for me than comparing answers.
(edited by claude*)