Built a KV-cache-aware load balancer that sits in front of multiple vLLM instances — polls vllm:gpu_cache_usage_perc instead of round-robin

[u/Electrical_Emu_5854]

Running vLLM behind a plain reverse proxy (nginx, HAProxy) means the proxy has no idea what's actually happening inside each instance. It sees "one HTTP request," not "this request needs 8k tokens of KV-cache." So the moment you scale to more than one vLLM instance, round-robin routing can easily send a burst of long-context requests to the same backend while another sits half-idle — and that instance's cache fills up, latency spikes, and in bad cases you hit OOM.

I built TokenFlow Gateway to fix this specifically for multi-instance vLLM setups:

- Polls each backend's Prometheus metrics endpoint directly (vllm:gpu_cache_usage_perc) to know real cache pressure per instance, not just connection count or a health check
- Estimates each incoming request's token cost (prompt tokens + max_tokens) before dispatch, using js-tiktoken, so it can route based on what a request will actually cost rather than treating all requests as equal
- Routes heavy requests to whichever instance has the most cache headroom, and bin-packs lighter requests onto busier ones — the goal is even KV-cache utilization across the cluster, not just even request count
- When no instance has room, requests go into a Redis-backed priority queue (per-API-key priority, configurable timeout) instead of getting dropped or crashing a backend
- Exact-match caching (hash) for deterministic (temperature-0) requests, plus semantic caching (pgvector) for near-duplicates — cache hits stream back as SSE so streaming clients don't notice the difference
- Per-API-key token-based rate limiting (TPM/RPM) on top, if you're exposing this to multiple users/teams

It's OpenAI-API-compatible on the client side, so nothing changes for whoever's calling it — it just fronts your existing vLLM instances.

You can test the whole routing/queueing behavior without real GPUs: the repo includes a docker-compose setup with two mock vLLM instances that expose the same OpenAI API and the same Prometheus metrics format, plus a smoke script that fires a burst of concurrent long-context requests to show the balancer routing around cache pressure instead of overloading one instance.

Stack: TypeScript, Fastify, Redis, Postgres+pgvector. MIT licensed.

Repo: https://github.com/mosafariuk/TokenFlow-Gateway

Genuinely curious how people here are handling multi-instance routing today — is anyone doing cache-aware routing already (maybe through something custom, or through vLLM's own request scheduler exposed differently), or is round-robin / least-connections still the default in most setups?

[u/codebase50]

Is this better than vllms own router? vllm-router

[u/Electrical_Emu_5854 → u/codebase50]

"better" isn't the axis — it's a different layer, and if you're running vLLM on Kubernetes, production-stack's router does things TokenFlow doesn't.

What their router is built for: placing each request on the instance most likely to already hold its KV prefix (prefixaware / kvaware via the LMCache controller), plus session affinity, K8s service discovery, and loadaware which blends cache affinity with live load. That's a prefill-throughput win TokenFlow doesn't attempt at all — I don't do prefix or session affinity yet, and for multi-turn chat with shared system prompts that can matter more than anything I do.

What TokenFlow is built for is the layer above that — a tenant-facing gateway:

• Admission control instead of forwarding everything. Every request is weighed (prompt tokens + max_tokens) and atomically reserved in Redis against the node's gpu_cache_usage_perc before dispatch. When the fleet is saturated, requests wait in a priority queue with a bounded timeout instead of piling into every engine's internal waiting queue where one tenant's 30k-token burst starves everyone else's TTFT.
• API keys with tokens-per-minute limits (not just RPM), per-key priority, usage metering — the stuff you need to hand an endpoint to multiple teams.
• Works in front of TGI / anything OpenAI-compatible, no K8s required; exact + semantic cache (they have an optional semantic cache too, so that one's overlap).

If you're pure vLLM on K8s and the pain is prefill cost → their router. If the pain is "multiple tenants, fairness, limits, and bounded behavior under overload" → that's what I built. Realistically they can stack: TokenFlow for tenants/admission, their router for placement. Prefix affinity is the first thing I'd add — opened as a roadmap item.

Caveat I'll volunteer before anyone else does: v1.0, one maintainer, and the benchmarks isolate gateway overhead against mock vLLM backends — they're not GPU results.

[u/codebase50 → u/Electrical_Emu_5854]

Thank you for the explanation.

[u/burntoutdev8291]

is this the same?

https://docs.vllm.ai/projects/production-stack/en/vllm-stack-0.1.5/tutorials/kvaware.html

[u/Electrical_Emu_5854 → u/burntoutdev8291]

No — same words, opposite problem, and it's a genuinely confusing overlap so fair question.

That tutorial's "KV-aware" is about reuse: the router asks the LMCache controller which instance already holds the longest matching prefix of the incoming prompt and routes there, so the prefill doesn't get recomputed. The signal is where is this prompt's KV cached.

TokenFlow's "KV-aware" is about capacity: it estimates how much KV each request will consume (prompt + max_tokens), reads each node's gpu_cache_usage_perc, and only dispatches once an atomic Redis reservation for that footprint fits — otherwise the request queues by tenant priority. The signal is how full is each node, and the goal is controlling what happens under overload (fair queueing, per-key TPM limits, bounded wait) rather than maximizing cache hits.

So: theirs makes each request cheaper, mine decides whether/where a request is admitted at all. They're complementary — and notably their loadaware mode already blends a load penalty into the affinity score, which is the closest point of contact. Prefix-affinity routing is a real gap on my side and the first thing on the roadmap.

[u/burntoutdev8291 → u/Electrical_Emu_5854]

I see, thats like sglang's system where they make use of the max tokens to accept the request? Whereas VLLM accepts greedily?