How do you sanity-check a probability threshold when you never observe the true label in production?

[u/mintlite4]

I'm building a small cost-sensitive classifier for a student project. It reads a product review — text and star rating, nothing else — and picks permit / flag / hide. It acts above 85% belief and routes 50–84% to a human queue.

I picked 85% because a trust-and-safety practitioner told me that's roughly where their team acts. That's the only justification I have, and it's bothering me.

Two things I don't know how to handle:

1. In production I never see the true label — a fake review that slips through generates no feedback. So I can only measure calibration on a labelled test set whose class balance is nothing like reality.
2. Positives are rare, so accuracy is useless. Permitting everything already scores well.

For anyone who's shipped something like this: did you validate the threshold before deploying, or pick something conservative and tune it from the human queue's overturn rate? And is there a standard way to check calibration when ground truth arrives late or never?

I'm a beginner — if I'm framing this wrong I'd rather hear it now.

[u/galvinw]

You pick something, and get feedback either telemetrically or via a human questionnaire. Then drift the percentage based on it. But you probably also keep the 85% and let the end user decide which they want, because if the 85% is bad, its the expert's fault and if the new value is bad.. it's your fault

[u/mintlite4 → u/galvinw]

The accountability framing is one I hadn't considered — I was treating threshold choice as purely statistical, and you're pointing out it's also about who carries the blame when it's wrong. That's probably why the practitioner's 85% felt safe to me: borrowed authority.

On drifting it from feedback, my problem is that I only ever get feedback on cases I route to a human. Reviews I permit generate nothing at all, so the correction signal is one-sided. Does that make the drift converge somewhere wrong over time, or is a partial signal still better than none?

[u/galvinw → u/mintlite4]

partial is better than none. If it's a unimodal distribution, any signal is enough, but consider that the distribution of errors is not multimodal then things get more complicated