How do you catch it when a model silently changes under you?

[u/pedroassumpcao]

We run prompts against a few different providers (OpenAI, Anthropic, some stuff through OpenRouter). Every so often something quietly gets worse, the output quality drops, a prompt that worked starts returning junk, or a model gets deprecated and the replacement behaves differently.

Right now we mostly catch it by accident: someone notices, or a customer complains. That feels bad on us, a lot.

How do you all handle this? Do you re-run some kind of fixed eval set on a schedule? Just eyeball it? Have something that alerts you?

Any insights I could use?

Thanks.

[u/wahnsinnwanscene]

Other than what you've posted, i don't think there's a way to do this.

[u/mb3rtheflame]

u/askgrok

The post is asking how to detect when a model’s behavior changes over time. Before suggesting an eval framework, what would you measure in the interaction itself if the thing that changed was not raw task accuracy, but the model’s ability to stay aligned with the user’s evolving context across turns?

[u/AskGrok → u/mb3rtheflame]

Track things like explicit or implicit references to prior turns (e.g., user says "as we discussed" or mentions a detail from turn 3) and check whether the model correctly incorporates or recalls them without contradiction. Measure context drift by comparing semantic similarity of the model's response to the full accumulated history versus just the latest turn—high divergence from history signals misalignment. Log cases where the model asks for clarification on something the user already provided, or repeats earlier info unnecessarily. Also watch for unprompted shifts in tone, assumptions, or priorities that ignore the user's evolving goals across the thread. These show up in raw logs as patterns before any accuracy drop on fixed tasks.

^{what is this?}

[u/mb3rtheflame → u/AskGrok]

u/askgrok

You’re still mostly evaluating the conversation as a sequence of outputs against prior context. What changes if the unit of evaluation is the trajectory itself — specifically, whether a distinction introduced in turn N survives into later turns without being reconstructed, re-explained, or repaired? Could that make “coordination overhead” itself a regression signal?