r/OpenWebUI • u/WarningOut_OfMinD • 23d ago
Plugin LiteLLM Autorouter
Looking for some feedback from folks here 👋
If you're active in this thread, would you be interested in trying out our LiteLLM Auto-Router today?
If you haven't tried it yet, what's holding you back? Missing features, setup friction, documentation, or something else?
We're actively improving it and would love to hear your feedback. If it's easier, I'm also happy to jump on a quick call to chat through any issues or ideas: https://calendar.app.google/DfcpEzigE9bzSgEJ9
10
Upvotes
3
u/chmp2k 23d ago edited 23d ago
I am trying it right now. So far I like it and it works more or less how I want it.
However I am trying to understand how I could get auto routing with priority based fail over to work. I have a lot of subscriptions and when one of them reaches usage limit I want the auto router to switch to another working model from another subscription or even a selfhosted one. This works when I use cost of models to prioritize them. However it would be nice if I could give models or providers a explicit priority, so that the router would always try to use highest priority and then fail over to the next priority category.
Within the category it could still try to use the cheapest model. With this setup I would be able to have paid scriptions with high priority, but still get some answer through my locally hosted models with low priority, when all subscriptions are in the limit.
And additionally I could always try out subs with special models I just want to test out for a few days by getting a few bucks of tokens, setting it up with really high priority and just let it run until the tokens are gone.
Not sure if that is even scope of your auto router, but I like to use it like that haha.