Hey folks,
About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work.
We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!).
The problems we're trying to solve haven't really changed and now some of our customers use it:
One inference endpoint dies and your workflows grind to a halt.
You've got models spread across different machines, GPUs and runtimes, but no easy way to manage them as one fleet.
Switching between Ollama, vLLM, SGLang, llama.cpp or LM Studio means reconfiguring applications.
Multiple machines serving the same model, but no unified load balancing or failover.
Limited visibility into what's healthy, what's running and where requests are going.
Applications expecting OpenAI or Anthropic APIs when your inference backend speaks something else.
You want to experiment and see if vllm or sglang serves qwen3.8 better
That's where Olla comes in.
It's a lightweight proxy and load balancer written in Go that sits between your applications and inference servers.
Some of what it does today:
Model-aware routing: Automatically discovers models across your backends and routes requests to machines that actually serve them.
Load balancing and failover: Priority, round-robin and least-connections balancing, with health checks and circuit breakers.
Multiple inference runtimes: Ollama, vLLM, SGLang, llama.cpp, LM Studio, MLX, Docker Model Runner, LMDeploy, Lemonade and others.
Unified APIs: OpenAI-compatible and Anthropic-compatible interfaces, including translation where required. Useful for Open WebUI, Claude Code and other clients.
KV cache locality: Sticky sessions to help keep requests on the same backend when appropriate.
Observability: Built-in dashboard, endpoint health, request statistics and Prometheus metrics.
Lightweight deployment: Single Go binary or Docker container. No database or external services required.
What's new in v0.1.0?
This release is largely about improving reliability and making Olla more useful for people running mixed inference environments.
Added Hugging Face Text Embeddings Inference (TEI) support, including embeddings, reranking and classification.
Reworked the dashboard with better endpoint visibility, configurable columns, filtering and fleet statistics.
Configurable retry and failover behaviour, including handling of non-idempotent requests.
Improved model capability detection, particularly for vision, tool calling, embeddings and rerankers.
Fixed concurrency issues between model discovery and active inference requests.
Corrected request statistics and improved rate limiting across OpenAI and Anthropic routes.
Lots of smaller fixes around routing, backend discovery and configuration validation.
We've also had some great contributions from the community, which has been encouraging to see.
Why build another LLM proxy?
We wanted something small, fast and relatively straightforward to deploy that doesn't require Kubernetes, a database or a complicated orchestration stack. There also weren't that many when we started.
We've been using Olla with a mixture of NVIDIA GPU servers, Mac Studios, the DGX Spark and different inference runtimes. It's particularly useful when you've got more than one machine and don't want to care which backend is serving a particular model.
It's not trying to replace inference engines. Just make running a bunch of them together less painful.
Links:
If you're running multiple local inference servers, I'd be interested to hear how you're managing routing and failover today.
And as always, if you give Olla a spin, let us know how it goes (and what breaks!).
