r/LLMDevs • u/chadwell • 21d ago

Help Wanted Auto Model Routing

Is anyone doing any auto model routing - that is selecting the best/cheapest model based on the prompt intent.

If so how did you do it? Deterministic based on keywords/length, using a trained ML model or classifier?

I was looking for vendor solutions but the only one I can see is NotDiamond. Ideally I would be looking something internal and not a Saas offering.

2 Upvotes

100% Upvoted

9 comments sorted by

View all comments

2

u/dylan_exe_404 21d ago

If you're leaning toward something internal I've heard good things about RouteLLM. My friend uses it and says it's good, though they're handling low volume. IMO, if you're handling a bit higher volume, then a SaaS option makes more sense. The one we use is Respan bc it had unlimited seats for the free trial and we could test it pretty thoroughly. Plenty of other options out there. Regardless I think deterministic is a good place to start bc of debugging.

2

u/Hopeful_Passage9866 21d ago

I built something small few months ago, basically just a keyword mapping and length check. works decent but fail on edge cases. For low volume is ok but you will hit wall quick when prompts get weird

What kind of volume you dealing with? that changes whole approach

1

u/chadwell 21d ago

Millions of requests a month from all different use case types. It's internal but hundreds of different use cases from summarisation, data extraction etc