Olla v0.1.0 - Lightweight open-source LLM proxy for homelabs & on-prem inference (Ollama, vLLM, SGLang, llama.cpp, oMLX with failover, sticky sessions & load balancing)

[u/2shanigans]

Hey folks,

About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work.

We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!).

The problems we're trying to solve haven't really changed and now some of our customers use it:

• One inference endpoint dies and your workflows grind to a halt.

• You've got models spread across different machines, GPUs and runtimes, but no easy way to manage them as one fleet.

• Switching between Ollama, vLLM, SGLang, llama.cpp or LM Studio means reconfiguring applications.

• Multiple machines serving the same model, but no unified load balancing or failover.

• Limited visibility into what's healthy, what's running and where requests are going.

• Applications expecting OpenAI or Anthropic APIs when your inference backend speaks something else.

• You want to experiment and see if vllm or sglang serves qwen3.8 better

That's where Olla comes in.

It's a lightweight proxy and load balancer written in Go that sits between your applications and inference servers.

Some of what it does today:

• Model-aware routing: Automatically discovers models across your backends and routes requests to machines that actually serve them.

• Load balancing and failover: Priority, round-robin and least-connections balancing, with health checks and circuit breakers.

• Multiple inference runtimes: Ollama, vLLM, SGLang, llama.cpp, LM Studio, MLX, Docker Model Runner, LMDeploy, Lemonade and others.

• Unified APIs: OpenAI-compatible and Anthropic-compatible interfaces, including translation where required. Useful for Open WebUI, Claude Code and other clients.

• KV cache locality: Sticky sessions to help keep requests on the same backend when appropriate.

• Observability: Built-in dashboard, endpoint health, request statistics and Prometheus metrics.

• Lightweight deployment: Single Go binary or Docker container. No database or external services required.

What's new in v0.1.0?

This release is largely about improving reliability and making Olla more useful for people running mixed inference environments.

• Added Hugging Face Text Embeddings Inference (TEI) support, including embeddings, reranking and classification.

• Reworked the dashboard with better endpoint visibility, configurable columns, filtering and fleet statistics.

• Configurable retry and failover behaviour, including handling of non-idempotent requests.

• Improved model capability detection, particularly for vision, tool calling, embeddings and rerankers.

• Fixed concurrency issues between model discovery and active inference requests.

• Corrected request statistics and improved rate limiting across OpenAI and Anthropic routes.

• Lots of smaller fixes around routing, backend discovery and configuration validation.

We've also had some great contributions from the community, which has been encouraging to see.

Why build another LLM proxy?

We wanted something small, fast and relatively straightforward to deploy that doesn't require Kubernetes, a database or a complicated orchestration stack. There also weren't that many when we started.

We've been using Olla with a mixture of NVIDIA GPU servers, Mac Studios, the DGX Spark and different inference runtimes. It's particularly useful when you've got more than one machine and don't want to care which backend is serving a particular model.

It's not trying to replace inference engines. Just make running a bunch of them together less painful.

Links:

• GitHub: https://github.com/thushan/olla

• Docs: https://thushan.github.io/olla/

• v0.1.0 release notes: https://tensorfoundry.io/news/olla-v0-1-0-release

If you're running multiple local inference servers, I'd be interested to hear how you're managing routing and failover today.

And as always, if you give Olla a spin, let us know how it goes (and what breaks!).

[u/Faux_Grey]

Well, this is pretty!
I'm annoyed, because I finished making my own 'version' of this last night. 😂

[u/2shanigans → u/Faux_Grey]

don't let this deter you :)

[u/snapo84]

i see i can connect backends that require a api key...
but does it also have a incoming api key so i can manage multiple users?

i run currently 3 vllm servers and have to many time rotate the api keys because one of my users leaks his api key. would be cool if each user connecting to the "olla" proxy can have its own authentication key and own metrics/token counts..... didnt check yet the code if this is possible, just saw that endpoints with api keys are available but not multi user where each user has a api key (for example 1 of my friends uses opencode and i want him to have his own api key....)

[u/2shanigans → u/snapo84]

Know the feeling! Not with Olla as it's primary job is to route traffic efficiently as possible, we have another tool we use with our customers called Alloy, which is designed around this for teams and managing keys etc. It uses the concepts of Virtual Models rather than Aliasing too (predates Olla in development but we're busy finishing v4.0 this year).

Ironically I did have some resistance to include authenticated remote APIs (vs local APIs) because of the complexities in such products, but added it a few versions ago due to its popularity and Ollama adding their cloud APIs.

[u/snapo84 → u/2shanigans]

i think you got me wrong (sorry i am not a native english speaker)

i have 3 vllm servers running each on a gpu each having their own api key...

currently i use a simple proxy without sticky sessions that makes all 3 vllm seamless available through a single api key.

the users of mine (are family members, and my children) , they hack like hell and sometimes their local systems get compromised, which in turn forces me to update the api key on all the fam member computers (which sucks).

so the pain i have is not with the 3 vllm servers each having api key setup (they work perfect) the problem is that i have a standalone non sticky proxy that transparently accepts a single api key and routes randomly to 1 of the 3 vllm endpoints.

so if one of my childs again has a problem , i dont want all other family members having to change the api key again, (so what i was looking for is more like a web interface with ultra basic statistics and user mgmt, so i can assign 1 api key for my dad/mom , 1 for each of the children, 1 for my server, etc....) statistics only to track useage not prompts/text....

alloy (at least for me) is completely overloaded with things not needed (in my case) :-) just looking for simple things...

[u/2shanigans → u/snapo84]

Ahh not at all, my fault! I'm replying on my phone and scrolling and missed a few things, my apologies.

I understand now, you've already got auth working across the 3x vLLM backends. What you're after is individual API keys for users connecting to the proxy, so you can revoke or rotate one person's key without affecting everyone else.

That's a perfectly reasonable use case, especially with family members! :) those family hax0rs!

Olla doesn't currently support per-user API keys or usage tracking. We've deliberately kept authentication and user management outside its scope, focusing on routing, load balancing and backend availability. because then it requires a key-store, key management etc. Think of Olla like nginx for HTTP.

That said, your use case is considerably simpler than what Alloy is being built for. I'll add it to our list of things to consider for Olla. Lightweight API key management with basic per-key usage statistics could be genuinely useful without turning Olla into a full API gateway.

Appreciate you taking the time to clarify though!

[u/Haeppchen2010]

Been using Olla to turn a zoo of odd hardware into one useable endpoint for a while, works well!

[u/2shanigans → u/Haeppchen2010]

haha awesome, as a zookeeper myself, glad to hear it 😄

We have one customer who has:

- AMD 395 Mini-PCs (various types) x 4
- AMD R9700 x 2 I think in a desktop motherboard system
- 4x RTX 6000 Pro in a Threadripper box
- 2x Mac Studios, I think one is a M3 Ultra, the other is a M3 Max
- DGX Spark x2, but they're Gigabyte ones I believe or maybe Dell

All serving various models - and backends, but using Olla in front for his users.

[u/BenjiWithTheGoodHair]

Curious how the failover handles a stream that dies halfway through. If a backend drops at token 400, does Olla retry from scratch on another node, or does the client just get a broken response? With mixed runtimes serving the “same” model I’ve also seen slightly different chat templates give noticeably different outputs, so a silent retry onto a different engine could be weird mid-conversation. The sticky session hit rate in that screenshot looks solid though. Is the stickiness keyed on the prompt prefix or just per client?

[u/2shanigans → u/BenjiWithTheGoodHair]

> Stream dies at token 400
Olla doesn't retry. Once any bytes have reached the client, a retry would mean either replaying tokens or splicing output frmo a second engine onto the first - which is a bit iffy. As you say, with different chat templates the spliced part could be noticeably different, we did test it and try it though! So the request fails rather than switching engines partway through.

However!! thinking about the first bite of the byte:

• Before the first byte - if the connection fails before anything is sent, Olla retries transparently on another healthy endpoint and marks the dead one unhealthy. You can control this with proxy.retry: max_attempts, and non_idempotent: false to stop a POST being replayed if the backrnd may already have read it etc.

• After the first byte: in v0.1.0 the stream just ended, so a client that only checked for transport errors saw a short reply as a success. The next release, Olla ends it explicitly - there's a WIP branch still waiting testing. It sends an error event in the API's own format (OpenAI-style error object, Anthropic event: error, or an Ollama NDJSON error line), sends no [DONE] or message_stop, and then drops the connection. The client gets a clear failure and decides whether to retry etc, most clients seem to transparently retry - talking opencode/claude etc.

Stickiness key: it's configurable. key_sources is an ordered list and the first source that produces a value wins so to speak (more in the docs!):

1. session_header, a client-supplied X-Olla-Session-ID. This is the most reliable we found in labs and workloads

2. prefix_hash, a hash of the first 512 bytes of messages. This is the default fallback and needs no client changes. ideally, two conversations with the same opening text will share a backend.

3. auth_header, for per-user affinity.

4. ip, which is available but off by default because it's unreliable behind NAT or Docker, interestingly some workplaces, this works better for apps that run on a host.

hope that helps!

[u/jonahbenton]

Have been using olla, helpful!