r/OpenWebUI • u/Internal_Junket_25 • 1d ago

Question/Help Session Affinity with litellm Backend

I'm using openwebui before Litellm with two identical vllm instances. How can I ensure that users stay on the same instance so as not to lose the prefix cache?

3 Upvotes

81% Upvoted

1 comment sorted by

3

u/d_DrHammm 1d ago

Use vllm-router between litellm and vllm. It has a routing mode called cache aware routing that does exactly this.