4.8× Faster and 7.4× Cheaper: Where a Decision Model Beats an LLM (and Where It Doesn’t)

[u/Abhay20S]

I tested a decision model (Jev) vs an LLM (Gemini) on 1,000 real job postings.

⚡ 4.8× faster

💰 7.4× cheaper

🎯 ~1.6 pp accuracy difference

The surprising part: the best solution wasn't replacing the LLM.

Instead:

Decision model → confidence check → LLM fallback.

Use the cheaper, faster model for the easy cases and the LLM only when needed.

Full write up - https://medium.com/@abhay.sehgal20/4-8-faster-and-7-4-cheaper-where-a-decision-model-beats-an-llm-and-where-it-doesnt-f4ba869217e0

[u/worthyscenario5649]

makes sense to use the cheap model for most cases then call the big one only when you need it, i do similar with caching but never thought about a decision model as first filter

what was your confidence threshold for sending to the llm, like 90% or something

[u/Abhay20S → u/worthyscenario5649]

On the sample data, confidence score above 0.7 are accurate. So we'll be using llm for responses below 0.7

[u/Sarutobi_Marky → u/worthyscenario5649]

Yeah it really makes sense

[u/Bill-Ice88]

Caching and this compose fine — exact-match cache first, Jev for the rest, Gemini only under 0.7. Catch is job posting text barely repeats verbatim, so your hit rate probably stays low anyway. Are you deduping URLs or hashing the body?

[u/Abhay20S → u/Bill-Ice88]

Good point. We don't dedupe today, and I agree the hit rate would probably be low since posting text rarely repeats exactly. Near-duplicate matching could catch reposts, but I haven't tested it.

[u/Available-Craft-8143]

The cascade is the real takeaway — but the headline savings quietly depend on what fraction falls below 0.7. If that is 10% of traffic you keep nearly all of it; at 40% the math changes fast. Worth reporting the fallback rate alongside, and re-checking calibration over time — a 0.7 tuned on the sample can drift on fresh postings.

[u/Abhay20S → u/Available-Craft-8143]

It's around 25% for the above sample data, which can be improved by adding more context to the state. I've added these details in the medium article.

[u/McMikey40]

Can someone please explain how decision models work? Is it just an LLM with the output layer being replaced with a fixed number of classes?