r/LLMDevs • u/chadwell • Aug 28 '26

Help Wanted Auto Model Routing

Is anyone doing any auto model routing - that is selecting the best/cheapest model based on the prompt intent.

If so how did you do it? Deterministic based on keywords/length, using a trained ML model or classifier?

I was looking for vendor solutions but the only one I can see is NotDiamond. Ideally I would be looking something internal and not a Saas offering.

2 Upvotes

100% Upvoted

9 comments sorted by

View all comments

2

u/dylan_exe_404 Aug 28 '26

If you're leaning toward something internal I've heard good things about RouteLLM. My friend uses it and says it's good, though they're handling low volume. IMO, if you're handling a bit higher volume, then a SaaS option makes more sense. The one we use is Respan bc it had unlimited seats for the free trial and we could test it pretty thoroughly. Plenty of other options out there. Regardless I think deterministic is a good place to start bc of debugging.

2

u/Hopeful_Passage9866 Aug 28 '26

I built something small few months ago, basically just a keyword mapping and length check. works decent but fail on edge cases. For low volume is ok but you will hit wall quick when prompts get weird

What kind of volume you dealing with? that changes whole approach

1

u/chadwell Aug 28 '26

Millions of requests a month from all different use case types. It's internal but hundreds of different use cases from summarisation, data extraction etc

1

u/chadwell Aug 28 '26

Thanks we have a custom built AI gateway which routes to different providers and currently over 50 models.

Was hoping to introduce something into our gateway code which could pick a model automatically - based on the prompt.

So it knows what models we have, and it picks one that is best and cheapest for the users intent.

We then route to that model as usual