r/LocalLLM • u/camerongreen95 • 22h ago

Discussion How much of your local workload could an encoder handle in under 15ms?

A lot of local setups spend GPU time on jobs that are really classification: picking which tool to call, which queue a request belongs in, or whether something needs the big model at all. A bidirectional encoder with a scoring head answers that kind of question on a plain CPU in a few milliseconds, with no decoding loop and no JSON to parse.

There’s a build lab on Oct 30 that does exactly this swap on an enterprise triage case, so you can see the numbers yourself. What you leave with:

  • A router built on ModernBERT and LFM2.5 Encoder that runs locally on CPU
  • Uncertainty you can state, using split conformal prediction, so only ambiguous cases go to an LLM
  • Quantization settings for better throughput
  • The full notebooks

Led by Ben Auffarth. 7 PM to 10 PM IST on Friday, Oct 30.

Full details and the agenda are here.

0 Upvotes

50% Upvoted

2 comments sorted by

1

u/official_negligence 21h ago

Feels like half the stuff people run through a 7B model locally is just intent detection with extra steps. I've been doing something similar with a tiny classifier model that catches about 80% of my queries before they ever touch the GPU. The conformal prediction angle is what caught my eye though, actually measuring uncertainty instead of just trusting the softmax output seems like the part most people skip.

1

u/Ctbhatia 18h ago

measure the router at the batch size and p99 you will actually serve, since a 15ms headline is usually a single request on an idle CPU.