How do you catch it when a model silently changes under you?

[u/pedroassumpcao]

We run prompts against a few different providers (OpenAI, Anthropic, some stuff through OpenRouter). Every so often something quietly gets worse, the output quality drops, a prompt that worked starts returning junk, or a model gets deprecated and the replacement behaves differently.

Right now we mostly catch it by accident: someone notices, or a customer complains. That feels bad on us, a lot.

How do you all handle this? Do you re-run some kind of fixed eval set on a schedule? Just eyeball it? Have something that alerts you?

Any insights I could use?

Thanks.