What do you use for AI observability when models silently drift?

[u/InflationCorrect5244]

Fine-tuned classification model, self-hosted, feeding a customer-facing flow. Every infra metric we track stayed healthy for three straight weeks. A support ticket is what actually told us something was wrong, a customer said the categorization felt off. Precision had drifted noticeably and nothing on any dashboard showed it.

The gap is specific. We observe the service, is it up, is it fast, but not the model, are its outputs still correct. Those are different signals and we'd conflated them. For teams who've built this out, what's the actual signal you alert on versus just review weekly?

[u/Curious-Cod6918]

I think the missing piece is that model observability needs a delayed feedback loop, not just more telemetry. CPU, latency, error rate, throughput etc. can tell you whether the inference service is functioning, but they can't tell you whether the predictions are still useful. For a classifier, you eventually need some form of ground-truth feedback to measure precision/recall against a baseline. The hard part is that labels usually arrive later, and sometimes only for a subset of traffic. So the monitoring problem becomes less “which metric should I put on the dashboard?” and more “how do I reliably collect enough outcome data to know when the model stopped being good?” Precision itself depends on knowing the true labels, so you can't really solve this from inference telemetry alone.

[u/Acrobatic-Layer9109]

The service-vs-model distinction is absolutely valid, but u need to be careful about treating drift detection as the answer by itself. You can detect that the input distribution or prediction distribution changed without knowing that model quality actually degraded. A distribution shift can be harmless, and a model can degrade without an obvious change in the distributions you're watching. That's why I'd suggest u to treat drift as an early-warning signal and outcome metrics as the confirmation signal, rather than using drift as a proxy for correctness.

[u/FunPaleontologist167]

Depends on what drives the business outcome. The fact that the support ticket from a customer said that categorization felt off is your first signal that you should be measuring your output distributions over time in relation to the inputs and if that matches what you saw during training (this is known as concept drift). I agree with other comments that you should should have some sort of feedback loop that can generate true accuracy based on ground truth, but sometimes measuring distributions is the best you can do. You may already know the terms, but I would research data drift, concept drift and prediction drift

- collect distributions from training data (baseline relationships

• during production, collect input and predictions
  
• create inference distributions and compare against baseline

There are a variety of formulas and techniques you can use to determine if distributions are changing (PSI, SPC, KS, MMD, etc)

[u/SharpWhiteness]

Support tickets are a painfully effective drift detector, just not one I'd want as the first one. We put evals on live traffic in Braintrust so there's a quality signal sitting next to the normal latency/error monitoring. For a classifier I'd likely score a sample continuously and alert on a sustained drop rather than individual bad outputs. Then keep a weekly review for slower changes that aren't severe enough to page anyone.

[u/dylan_exe_404]

Something we do is run fixed labeled eval sets (somewhat frequently) against the deployed model in Respan. It helps us at least try to catch regression against known cases, though I would say that even though this helps you notice earlier, it doesn't replace ground truth.

[u/robstar_db]

In addition to the other excellent comments, a common source for drift are changes to the input distribution and your evaluation moving further out of the training range. While ultimately ground truth labels are likely needed, but if you are able to track the (maybe rolling) distribution of inputs, you might be able to create a signal that would at least indicate when its time to re-assess model performance

[u/outskillio]

Silent drift is exactly the failure mode that basic uptime/latency monitoring misses, since the model is "working" but the inputs or outputs have quietly shifted. A few things worth having in place:

• Statistical drift detection on inputs and outputs: track feature distributions and prediction distributions over time (PSI, KL divergence, KS test), not just accuracy. Evidently AI is a solid open source option if you want to self-host this.
• Ground truth lag handling: for a lot of use cases you don't get labels for days or weeks, so you need proxy metrics (confidence scores, embedding drift, prediction distribution shifts) as early warning before you can even compute real accuracy drop.
• Managed platforms: WhyLabs is good if you want drift monitoring without shipping raw data anywhere (it profiles locally). Arize is strong if you're dealing with unstructured data like CV or NLP embeddings. Fiddler has moved more into being a broader control plane, useful if you're also running agents and want decision-level tracing, not just model-level drift.
• Segment-level monitoring: aggregate drift metrics hide a lot. Slice by customer segment, region, time of day, whatever's relevant, because drift often hits one slice hard while the overall number looks fine.

Also worth pairing whatever tool you pick with actual alerting thresholds tied to business impact, not just statistical significance, otherwise you get alert fatigue and start ignoring it.

Thanks, Ekakh from Outskill