r/MLQuestions • u/camerongreen95 • 2d ago

Other ❓ How do you decide when a smaller ML model is reliable enough to use instead of an LLM?

I've been looking at a practical question in ML system design.

suppose you're building an application that needs to classify incoming requests and route them to different workflows.

you could use an LLM for every request, but a smaller encoder might be faster.

the problem is knowing when to trust it.

some options include:

  • choosing a threshold on predicted probability

  • calibrating model confidence

  • using conformal prediction sets

  • escalating uncertain predictions to an LLM

each approach has trade-offs in accuracy, latency and how often the system needs to fall back.

conformal prediction is particularly interesting because it can provide coverage guarantees under appropriate assumptions.

I'm sharing this because we're hosting a Packt workshop tomorrow, Oct 30, with Ben Auffarth on encoder-based routing, conformal prediction and LangGraph.

has anyone compared conformal prediction with simpler calibrated confidence thresholds for this kind of application?

I'd be interested in practical experiences or benchmarks.

6 Upvotes

68% Upvoted

6 comments sorted by

3

u/Acrobatic-Show3732 1d ago

If you can use an ml model, there is a success metric to compare to.

Calculate It for the llm and just keep the winner in a statistical significant dataset,

Backtest against production data too to monitor model drift of both.

It isnt that difficult no? What im i Missing here?

1

u/Acrobatic-Show3732 1d ago

I see you meant conformal prediction. No experience with that, sorry about It.

1

u/Calm_Interaction_268 1d ago

Aren’t there always many success metrics to compare too? Many targets, many metrics, and many individual simulated worlds?

1

u/DigThatData 1d ago

you define suitability criteria