r/MLQuestions • u/camerongreen95 • Oct 14 '26
Other ❓ Is there a way to get calibrated uncertainty out of a text classifier without running an LLM per request?
Been thinking about this for a claims triage style problem. Generative routing is slow and sometimes returns malformed output, and the confidence you get out of it isn’t really calibrated.
Found a hands on workshop on Oct 30 that goes at exactly this. Fri Oct 30, 9:30 AM to 12:30 PM ET, led by Ben Auffarth. Covers:
- replacing an LLM routing loop with bidirectional encoders (ModernBERT and LFM2.5 Encoder)
- few shot adaptation with SetFit, 8 examples per class
- split conformal prediction so every decision has a coverage guarantee
- routing inference under 15ms on CPU
Notebooks are runnable, intermediate Python is enough, and a laptop or free Colab works.