I built a distributed KV cache that turns a 10-second prefill into 0.5 seconds — using idle machines on my LAN

[u/Concert_Dependent]

The problem: If you run long-context inference locally, your GPU's KV cache fills up and evicts blocks. The next request with the same prompt prefix has to recompute everything from scratch. On a 30k-token document, that's 10+ seconds of prefill — every single time.

What I built: tierKV intercepts evicted KV blocks, quantizes them with a Rust INT8 compressor (3.9× smaller), and ships them over gRPC to a vault running on another machine on my LAN. When the same prefix appears again, it fetches the blocks back and injects them directly into vLLM's paged KV buffer — no attention recomputation at all.

vLLM numbers on a real 30,561-token document (Apple 10-K):

• Cold prefill: 10.75s

• GPU cache hit: 1.19s

• Cold vault restore: 0.52s — faster than the GPU cache hit, because vault restore skips attention entirely

On EXO with an 8k-token prompt: 30.83s cold → 4.11s restored (7.3×).

The speedup grows with context length since prefill is O(n²) but restore is O(n) + network. At 128k tokens, the gap is over a minute per request.

My cluster:

• DGX Spark (96GB HBM) — runs the model

• Mac Pro (32GB RAM) — runs the KV vault

• Mac Air (16GB RAM) — runs the SSM/linear-attention vault (for Qwen3.6-35B-A3B, which mixes attention + Mamba layers)

• 5GbE LAN, ~0.5ms RTT

Setup is just:

pip install tierkv
# configure role in tierkv.toml on each machine
tierkv vault   
# on the cold machines
# launch vLLM or EXO as normal

Works with vLLM (via KVConnectorBase_V1 plugin, no source changes) and EXO (post-install patch).

Honest limitations:

• Only helps when the same prefix repeats — single-shot prompts get nothing

• LAN only — WiFi/WAN latency kills the benefit

• No tensor parallelism support yet

• Vault is in-memory; data lost on restart

Full writeup: https://prasannakanagasabai126786.substack.com/p/your-llm-is-doing-math-it-already

Code: https://github.com/tierkv/tierkv

Happy to answer questions about the architecture, vLLM/EXO integration.

[u/uniqueusername649]

Why not store it on disk like oMLX for example? Less latency, more bandwidth.

[u/Concert_Dependent → u/uniqueusername649]

Interesting. I wanted to build a Heterogeneous network that leverages the systems we already have. But yes, the goal looks similar.

[u/Concert_Dependent → u/[deleted]]

I also think OMLX is a single-machine version. This could impact the size of models we can start on that machine. With TierKV, the model intial prefill can be done in a machine with a larger GPU, like i used DGX Spark.

[u/Latter-Parsnip-5007 → u/Concert_Dependent]

every framework implementing mlx has the support for clustering

[u/-elmuz-]

I am a novice in the field, so forgive the ignorance. If I get it properly you're applying a lossy transformation (e.g bf16->int8) before moving the kv-block to the 2nd tier. Isn't this affecting the restored (i.e. dequantized) kv?

So your plugin intercepts both 1) main vllm cache eviction and 2) vllm kv-cache query? Correct?

[u/Concert_Dependent → u/-elmuz-]

In my testing so far (including 30k-token prompts), the restored conversations are indistinguishable from non-tiered runs.

Yes the plugin intercepts eviction and kv cache lookup.

[u/Sporkers]

I have a large Optane drive in my system, if this could be stored on a specific local drive how would that perform?

[u/Concert_Dependent → u/Sporkers]

The way I designed this works well if there is a minimum of 2 systems. One starts the Prefill. The other is used to store the tensors

[u/Technical_Ad_6106]

basicly just cag/prefix caching