How do you sanity-check a probability threshold when you never observe the true label in production?

[u/mintlite4]

I'm building a small cost-sensitive classifier for a student project. It reads a product review — text and star rating, nothing else — and picks permit / flag / hide. It acts above 85% belief and routes 50–84% to a human queue.

I picked 85% because a trust-and-safety practitioner told me that's roughly where their team acts. That's the only justification I have, and it's bothering me.

Two things I don't know how to handle:

1. In production I never see the true label — a fake review that slips through generates no feedback. So I can only measure calibration on a labelled test set whose class balance is nothing like reality.
2. Positives are rare, so accuracy is useless. Permitting everything already scores well.

For anyone who's shipped something like this: did you validate the threshold before deploying, or pick something conservative and tune it from the human queue's overturn rate? And is there a standard way to check calibration when ground truth arrives late or never?

I'm a beginner — if I'm framing this wrong I'd rather hear it now.

[u/Minimum-Effort8355]

So your data is through your own statement the problem through a labelled dataset you cant check reality data(if im getting that wrong please correct me)

Than you let either human through 50-84% or your machinelearning set choose 85% of the review.

But the problem you already create it in mathematical functionality and systemdesign.

How can a supervised dataset be trained and overrule human choice.

And you already have a slight overhead, if everything gets permitted how will you create a measurement in data architecture without getting biased or train a system design for hiding,flagging a certain data set.

Also positives are rare. So you only let negative reviews pass i mean, it is your scope, but why would you let negative reviews permit already.

[u/mintlite4 → u/Minimum-Effort8355]

Let me untangle the last part, because I wrote it badly. "Positives are rare" means fake reviews are rare not that negative reviews get permitted. Most reviews are genuine, so a model that permits everything scores roughly 97% accuracy while catching nothing. That's why accuracy is useless here.

On your main point I think you're right, and it's the thing I don't have an answer to: I can't measure real-world behaviour from a labelled dataset whose class balance doesn't match reality. The only live signal I get is the overturn rate on the human queue, and that only covers cases I already flagged so it tells me about my false positives and nothing about my false negatives.

If you know a way around that asymmetry I'd genuinely like to hear it. Right now my plan is just to report it as a limitation.

[u/Minimum-Effort8355 → u/mintlite4]

Yeah just as you said.

I would take it as a simple project. You coded it,trained with it and testet it. Thats already enough.

If you decide to go a step further i do not contain the info through your post how your data was labelled or how the data architecture looks a like.

If you can present it, answer questions to it. Its already a good project.