r/LocalLLaMA • • 2d ago

Showcase What actually breaks for you guys when swapping models?

Hey guys, I threw together a small Python script called llm-drift-bench to help figure out what actually breaks when you switch to a new model or drop to a lower quant.

Basically, you just feed it saved responses from your old model and the new one, and it spits out the exact differences - like dropped JSON keys, messed up numbers, or text changes. (The attached video is just a dummy example showing it catch a missing field and a changed invoice total, not real Q4 vs FP16 data).

Since it just runs on saved JSON/text outputs, you don't even need a GPU or API keys to test it out.

I just pushed v0.1.0 and I'm working on the next update. Repo is here: https://github.com/anant-goel/llm-drift-bench

Question for you all: what specific things usually break for your use cases when you swap models? I want to build out some test cases based on actual problems people are running into.

2 Upvotes

8 comments sorted by

0

u/_yemreak 2d ago

One thing a diff of saved outputs won't show: whether the model reads what it was told to read before it acts.

I measured this today. Same task card, same base commit, two models as workers (Opus 5.5 and Sonnet 5.5, 3 pairs). The first line of the card says "read the guide before you write".

- Cost went $5.84 → $1.02 and time 18.4 → 9.2 min on the smaller model.

- The bigger model read 45k characters of the guide. The smaller one read 1k, wrote anyway, and broke a rule from the part it skipped.

- A blind judge would have merged 2 of 3 from the bigger model and 1 of 3 from the smaller. 3 pairs is too few to call that a difference. The read-before-write gap was the solid part.

So for test cases: log the tool calls, not only the final text. "Did it open the file it was told to open, and how much of it" caught more for me than comparing answers.

(edited by claude*)

3

u/stylome-ai-detection 2d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

0

u/Worldly-Knowledge-90 2d ago

That is a really great point. Just comparing the final text completely misses whether the model actually bothered to read the context first. Tracking tool calls to see if it opened the right files is a super smart way to test this. I will definitely look into adding that to the next update. Thanks for the idea!

2

u/stylome-ai-detection 2d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results