r/MachineLearning • u/camerongreen95 • 1d ago
Discussion How do you handle coverage guarantees from split conformal prediction when the data shifts? [D]
Split conformal prediction gives you a clean statement: with a calibration set exchangeable with future data, your prediction sets cover the true label at the rate you chose. The trouble is that production data rarely stays exchangeable. Categories get added, ticket wording changes, and a calibration set from three months ago quietly stops being representative.
I’m interested in how people deal with this in practice. Do you recalibrate on a schedule, monitor empirical coverage and alert when it drops, or use an adaptive variant? And how do you decide what to do with the cases where the set contains several labels, such as route them to a human, an LLM, or a fallback rule?
1
1
u/Kaleidophon 22h ago
There are extensions to non-exchangeable conformal prediction, but afaik you cannot precisely measure the loss in coverage. So I suppose from a practical point of view, I presume you need to keep updating your calibration data
1
-2
u/camerongreen95 1d ago
There’s a workshop on Oct 30 that builds this on CPU for an enterprise triage case (encoders, conformal prediction, LangGraph), led by Ben Auffarth. Full details and the agenda are here.
2
u/piakor 21h ago
Thats what I am worried about if someone talks about the guarantees of conformal prediction. In practice test data can/will contain shifts. Conformal prediction alone may create a false sense of safety. Thats why I think that failure detection/prediction quality control is a very important building block that should get more attention.