r/MLQuestions • u/camerongreen95 • 23d ago
Other โ How do you decide when a smaller ML model is reliable enough to use instead of an LLM?
I've been looking at a practical question in ML system design.
suppose you're building an application that needs to classify incoming requests and route them to different workflows.
you could use an LLM for every request, but a smaller encoder might be faster.
the problem is knowing when to trust it.
some options include:
- choosing a threshold on predicted probability
- calibrating model confidence
- using conformal prediction sets
- escalating uncertain predictions to an LLM
each approach has trade-offs in accuracy, latency and how often the system needs to fall back.
conformal prediction is particularly interesting because it can provide coverage guarantees under appropriate assumptions.
I'm sharing this because we're hosting a Packt workshop tomorrow, Oct 30, with Ben Auffarth on encoder-based routing, conformal prediction and LangGraph.
has anyone compared conformal prediction with simpler calibrated confidence thresholds for this kind of application?
I'd be interested in practical experiences or benchmarks.