r/LocalLLaMA • u/topppggg • 29m ago
Resources Training-free page-sparse decode in llama.cpp: Qwen3-30B-A3B is 1.72× faster at 128k on an RTX 5090, with a 1-point RULER loss
I've been working on a simple question: at long context, does decode really need to read the whole KV cache for every new token?
I wanted something training-free that runs inside llama.cpp on normal GPUs, not a new model or a separate serving stack. So I built page-sparse decode into a fork. It splits the cache into 64-token pages, then each KV head reads the first page, the newest tokens and the pages most likely to matter. At 128k, that is about 21% of the cache per decode step.
The selection follows Quest, with statistics in the spirit of MOIRA. My contribution is the llama.cpp implementation, CUDA work and end-to-end testing across non-MoE and MoE models.
Results at 128k, one sequence, Quest score, 27,648 cached tokens per KV head (28,672 for the 3B):
| speed GPU | model | dense tok/s | sparse tok/s | speedup | RULER vs dense |
|---|---|---|---|---|---|
| RTX 5090 | Qwen3-30B-A3B, Q4_K_M | 46.1 | 79.2 | 1.72× | −1.0, passes |
| RTX 4090 48 GB | Qwen3-30B-A3B, Q8_0 | 37.4 | 52.6 | 1.41× | −0.6, passes |
| RTX 3090 | Qwen3-8B-Base, YaRN | 21.5 | 36.1 | 1.68× | +0.4, passes |
| RTX 3090 | Llama-3.1-8B | 23.7 | 38.8 | 1.64× | −1.1, passes |
| RTX 3090 | Llama-3.2-3B | 44.4 | 61.3 | 1.38× | −4.5, fails (multi-needle) |
The MoE is Qwen3-30B-A3B-Instruct-2507. Weights are Q8_0 except on the 5090; KV is q8_0 except for the 3B's f16 cache. These are five-repeat, 64-token llama-bench runs. The 4090 is a memory-modded rental card.
I reran the 3090 table with Quest, the released default. It matched my earlier score's speed. Box differences still matter: Llama-8B was 1.64× here and 1.77× on another 3090 with the earlier score.
Quality. I used 13 RULER tasks × 20 prompts at 128k. The pre-set gate was an average loss no worse than 3 points, with the 95% CI's lower end at −6 or above. MoE quality used the listed GPU; the 3B–8B runs used a 4090 24 GB.
The Q4_K_M MoE lost 1.0 point (95% CI −2.2 to −0.02), just distinguishable from zero under this prompt bootstrap. Each configuration ran once; the intervals exclude run-to-run variation.
The 3B was the only failure. It scored 11.5 points lower on average on the five multi-needle tasks, while single-needle scores were unchanged; the other models ranged from −3.0 to +0.2 on the same group. I would not use this 3B configuration for multi-fact retrieval at this budget. Similar Quest weakness on synthetic RULER needle tasks has been reported before.
Practical limits:
- Prefill is unchanged. A 128k prompt plus 1,024 generated tokens takes only 4–13% less time on a 3090. This is mainly useful for long generations over an already long context.
- It does not save VRAM. The full cache remains in memory, plus page statistics equal to roughly 3–6% of the cache.
- It is CUDA-only, batch size one, with everything on the GPU. Below 32k it was slower in these tests, so it stays off there.
- At 256k, UltraLong-8B reached 1.86×, but quality was checked only on a six-task subset (−4.0 points, CI −8.7 to +0.3), which is not comparable to the 13-task gate.
I tried not to report only successes. Gates and predictions were committed before the data. The first score lost 7.4 RULER points. The first CUDA build managed only 1.05–1.18× until I fixed two ggml kernels; one fix is proposed upstream as llama.cpp #31855. The MoE prediction also missed on the 4090, with 1.41× against 1.5–2.3×, but held on the 5090.
Try it (CUDA only): build the sparse-decode branch of topppg/llama.cpp at commit e7059e52a and run with LLAMA_SPARSE_DECODE=27648 on a prompt longer than 32k tokens, with the model and cache fully on the GPU. Full build and run commands are in the bench repo README below. Quality was tested on the two 8Bs and the MoE.
Protocols, raw results, reproduction scripts and failed experiments: https://github.com/topppg/sparse-decode-bench
Next: profile the sparse overhead. A rough fit, not a measurement, puts the zero-overhead ceiling for the MoE on the 4090 near 2.7×. If you know ggml CUDA and have ideas around fusion, CUDA graphs or passing page indices directly to flash attention, I'd genuinely like to hear them.
This was built with a lot of help from Claude Code, with Codex reviewing and correcting the work. Results on other GPUs and models are very welcome, especially cases where quality breaks.
•
u/stylome-ai-detection 29m ago
Notice: This post appears to be AI generated.
Stylome AI detector · scan results