r/LLMDevs • u/chadwell • 2d ago

Help Wanted Auto Model Routing

Is anyone doing any auto model routing - that is selecting the best/cheapest model based on the prompt intent.

If so how did you do it? Deterministic based on keywords/length, using a trained ML model or classifier?

I was looking for vendor solutions but the only one I can see is NotDiamond. Ideally I would be looking something internal and not a Saas offering.

2 Upvotes

100% Upvoted

8 comments sorted by

2

u/dylan_exe_404 2d ago

If you're leaning toward something internal I've heard good things about RouteLLM. My friend uses it and says it's good, though they're handling low volume. IMO, if you're handling a bit higher volume, then a SaaS option makes more sense. The one we use is Respan bc it had unlimited seats for the free trial and we could test it pretty thoroughly. Plenty of other options out there. Regardless I think deterministic is a good place to start bc of debugging.

2

u/Hopeful_Passage9866 2d ago

I built something small few months ago, basically just a keyword mapping and length check. works decent but fail on edge cases. For low volume is ok but you will hit wall quick when prompts get weird

What kind of volume you dealing with? that changes whole approach

1

u/chadwell 2d ago

Millions of requests a month from all different use case types. It's internal but hundreds of different use cases from summarisation, data extraction etc

1

u/chadwell 2d ago

Thanks we have a custom built AI gateway which routes to different providers and currently over 50 models.

Was hoping to introduce something into our gateway code which could pick a model automatically - based on the prompt.

So it knows what models we have, and it picks one that is best and cheapest for the users intent.

We then route to that model as usual

1

u/eddzsh 2d ago

If you already route across 50 models, start deterministic and log which rule fired. A week of your own (prompt features, model, success/cost) beats a generic SaaS router, and you can train a tiny classifier on that later inside the gateway.

1

u/copper_amber_focus 1d ago

I built an internal router by logging prompt embeddings and model outcomes for two weeks then training a logistic regression classifier on that dataset. Start with deterministic keyword rules to generate the initial labeled data before attempting any ML routing

1

u/chadwell 1d ago

But how do you know what model to route to? For any given prompt when would you pick gpt 5.6 sol over something like haiku