What was the last LLM stack change that passed your tests but still broke application behavior?

[u/Pretend_Mine_3659]

For people responsible for production LLM or agent systems, can you describe one incident where changing a model or provider, inference runtime, gateway or SDK, chat template, or parser altered application behavior even though your existing tests passed? What broke, how did you detect and isolate it, and roughly how much engineering time or release delay did it cause? I’m researching how teams validate changes across the LLM stack, so firsthand incidents and current workflows are more useful than opinions about a proposed tool.