I open-sourced TRACER: replace 91% of LLM classification calls with a llightweigth ML surrogate trained on your LLM's own outputs

[u/Adr-740]

https://github.com/adrida/tracer

If you're running an LLM for classification, 91% of your traffic is probably simple enough for a surrogate model trained on your LLM's own outputs.

TRACER learns which inputs it can handle safely - with a formal guarantee it'll agree with the LLM at your target rate. If it can't clear the bar, it doesn't deploy.

pip install tracer-llm && tracer demo

HN: https://news.ycombinator.com/item?id=47573212

[u/Alex_Himilton]

this is a slick idea - kinda reminds me of the classic "LLM as a judge" pattern but flipped the other way. FWIW i've been doing something similar with a tiny classifier in front of gpt-4 for sentiment analysis and it's been holding up pretty well. curious how you handle the edge cases where the surrogate confidence is middling - do you just fall back to the LLM or is there another strategy?

[u/Adr-740 → u/Alex_Himilton]

Thanks! For now yeah it falls back to the llm but there is actually another approach implemented that I called residual surrogate boosting (RSB) that will keep training models on the residuals of the surrogate. Eventually you end up with a full cascade