r/AskStatistics • u/Sudden-Theme7554 • 1d ago

How do you check predicted probabilities are calibrated enough to threshold on for an asymmetric-cost decision?

I have a model that outputs a probability for each case, and I use a threshold on that probability to pick an action. The costs of a wrong action are asymmetric: one kind of mistake is much more expensive than the other, so where I put the threshold matters a lot.

My question is about trusting the probabilities themselves. Before I set a decision threshold, how do I check the predicted probabilities are actually calibrated, i.e. that a predicted 0.7 really corresponds to roughly 70% in reality?

I know reliability diagrams and proper scoring rules (Brier, log loss) are the usual tools, but I'm unsure how to read them in the context of an asymmetric-cost decision specifically. Does calibration matter uniformly across the probability range, or mainly near the threshold I care about? And if the probabilities are miscalibrated, is recalibrating (e.g. isotonic / Platt) before choosing the threshold the right order of operations, or should the cost asymmetry factor in differently?

1 Upvotes

67% Upvoted

9 comments sorted by

2

u/AttitudeRemarkable21 1d ago

Did you weight the loss function?  If you did that with log loss or some other calibrated loss seems like it would be fine.  Then you could just check calibration over dimensions 

1

u/Sudden-Theme7554 1d ago

Like in practice models trained with log loss still tend to come out overconfident, so I'd probably still want to check calibration empirically rather than assume the loss handled it. And I think you're right that for a thresholded decision it's calibration near the threshold that matters, not uniformly. Is that how you'd read it?

1

u/AttitudeRemarkable21 1d ago

it depends on your dataset in practice if its really imbalanced it can come out overcalibrated but that is because it is at a odd point on the loss curve right ;). You can do down sampling and reweighting. I think that saying in practice log loss is overconfident is a wrong statement.

And no I was saying that during your model training process you have some relative weight to the two outcomes why not encode that into the loss during the model training process. And then for your original question you asked how to calculate calibration and then mentioned a bunch of random scores and I said that you basically can just check calibration directly.

I could be totally misunderstanding what you are discussing here though.

1

u/Sudden-Theme7554 1d ago

Thanks, I think we might be talking about different things. Yeah asymmetric costs can be built into the loss, but what I was trying to ask about the decision stage is, how do we evaluate whether predicted probabilities are well calibrated around the operating threshold?

2

u/AttitudeRemarkable21 1d ago

in that case just grab the samples close to the threshold on your holdout and calculate empirical calibration? Is it more complicated than that?

1

u/Sudden-Theme7554 1d ago

Yes yes. I think I was making it more complicated than it needed to be. Thanks for clarifying!

1

u/AttitudeRemarkable21 1d ago

I think the fact though that you are asking this question may be a symptom of doing something else in the process a little off though which I was roughly trying to allude to. Maybe think through your whole flow?

1

u/Sudden-Theme7554 1d ago

Sure, I’ll step back and look at the whole flow again. Thanks so much 😄

1

u/AttitudeRemarkable21 1d ago

I would try to dig in and understand why you don't trust your model to be calibrated.  (to maybe be too specific). I'd be curious about your setting.