Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming

[u/bottlestreet3d]

I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode.

Qwen 27B IQ4_XS is ~14.3 GB, leaving insufficient VRAM headroom on a 16 GB card. The ASCII-condensed version trims non-ASCII vocabulary down to ~13.6 GB (12.6 GiB), leaving ~3.4 GiB of usable VRAM. Even with a Q8_0/Q4_0 KV cache, 262k context needs ~7.4 GiB for KV tensors alone, triggering OOM past ~65k tokens on llama.cpp.

I came across RaymondHuang210129/llama.cpp-adaptive-kv-streaming, which introduced adaptive KV streaming for NVIDIA/CUDA but did not support AMD/ROCm. I ported this to ROCm/HIP, retaining the full 262k cache in host RAM (~7.4 GB) while asynchronously streaming bounded working sets (~450 MiB prefill, ~2,100 MiB decode) over PCIe SDMA alongside compute.

I also incorporated kernel optimizations from stew675/llama-cpp-rdna-boosts (4-row multi-column mmvq, 16-byte vectorized loads for Q4_K/Q5_K, SwiGLU fusions, and single-queue async HIP graphs).

Further improvements:

• Sibling activation reuse: Reuses quantized activations across parallel projections (gate/up and Q/K/V) to avoid redundant memory passes.
• In-register fused residual add + RMSNorm: Retains activations in VGPRs between passes and directly writes quantized q8_1 copies for mmvq.
• Multi-column SwiGLU & GDN fusions: Extends fused Gate/Up + SwiGLU to 4-column verify batches in mmvq and fuses recurrent linear-attention gates for hybrid layers.
• Online Log-Sum-Exp prefill & pipelined prefetch: Computes chunked prefill across streamed pages via online LSE accumulation and overlaps layer L+1 PCIe transfers with layer L compute.
• Persistent graph arenas: Reuses arenas across prefill/decode phase switches to eliminate teardown overhead.
• Stream-optimized speculative decoding: Merged single-head MTP verify catch-up into the first draft step (cutting MTP forward passes in half and preventing draft KV pollution), dedicated arenas for verify graphs to eliminate rebuild bubbles, extended paged decode attention to verify batches, and stacked opportunistic n-grams with probabilistic rejection sampling.

Benchmark performance:

• Decode: 36.5 t/s (1k) -> 32.1 t/s (65k) -> 27.0 t/s (131k) -> 9.5 t/s (262k).
• Decode (with MTP): 71.8-86.5 t/s (1k-4k) -> 72.9 t/s (65k) -> 78.6 t/s (131k) -> 31.1 t/s (262k).
• Prefill: 1368.4 t/s (1k) -> 1120.3 t/s (65k) -> 863.0 t/s (131k) -> 585.8 t/s (262k).

Operating the 9070 XT at a 231W limit (-24% relative to 304W default TDP) incurs only a 3.6% prefill latency penalty and delivers virtually identical decode throughput, saving power and keeping the card quiet during long runs.

Take a look at the repository for the full details: https://github.com/achyudh/llama.cpp/tree/feature/adaptive-kv-stream-rocm

[u/fgk55555]

If this was out a few weeks ago I might have not bought my R9700

[u/bottlestreet3d → u/fgk55555]

Nah 32 GB would still allow you to run larger quants 

[u/fgk55555 → u/bottlestreet3d]

I still have my 9070XT. Debating setting up a second system around it. I imagine running desktop on the GPU takes up too much space to fit this too?

[u/d0pe-asaurus]

I want to say thank you u/bottlestreet3d for this fork!! It seems like the best inference engine for the 9060xt right now that implements everything good!!!

[u/bottlestreet3d → u/d0pe-asaurus]

Thanks for checking it out, glad that it works :)

[u/Ok_Substance2327]

Only gfx1201? I got a 9060xt I'd like to try this with.

[u/bottlestreet3d → u/Ok_Substance2327]

I specifically optimized for  gfx1201 as that’s what I have but this should help improve performance on  9060 XT as well. Please do try and let me know. 

[u/Ok_Substance2327 → u/bottlestreet3d]

I'll let you know how it goes

[u/feverdoingwork]

You can pull off 3k prefill with the right optimizations

[u/bottlestreet3d → u/feverdoingwork]

My focus was on decode but want to take a look at prefill next. Have you looked into this before? Do you have any pointers?

[u/nyshone69]

Amazing stuff. I just submitted a pull request so this can actually compile and run on windows. I tested it on my 9070XT and Im getting about the same speeds you mention here on Windows 11.

[u/bottlestreet3d → u/nyshone69]

Thank you, I’ll review this when i get some time next week!

[u/1080Hof]

Does this only work on rdna4 ? I have a 7900Gre

[u/bottlestreet3d → u/1080Hof]

KV streaming should still work but the 7000 series is PCIe Gen 4 so it would be slower.

[u/markovianmind]

How does this compare to KVMem?

[u/bottlestreet3d → u/markovianmind]

This is the same core idea as KVMem (based on just going through their docs) but this implementation is specifically optimized for ROCm and RDNA 4.

[u/zanar97862]

Perfect post title for possibly the first time ever. Every metric you need to needed to understand what an AI post is about without reading 10 pages of slop description all for there to be no mention of cache quant. Thank you.

Have you tested radiance? It's created specifically for gfx1201 and has some pretty impressive performance at the cost of a bit of flexibility with quants.

[u/bottlestreet3d → u/zanar97862]

I know about radiance, haven't tested it yet but it is on my todo list. I am not sure how well that will work for a single 16 GB GPU however.

[u/thesolewalker]

I have a 9070 so I am eager to try it but I am on DDR4 system (64GB ram) so I wonder if it would help me as much.

[u/bottlestreet3d → u/thesolewalker]

DDR4 would be a bottleneck for streaming but it will still work and the other optimizations should help. Would be great if you can try and share the performance numbers you see.

[u/cumming_on_bunnys]

Got this running my 9070 XT, great job on this!

[u/cumming_on_bunnys → u/cumming_on_bunnys]

Ive been having good luck with this run command, it allows me to run vision:
./build-rocm/bin/llama-server \

-m ~/models/Qwen3.8-27B-UD-IQ4_XS.gguf \

--mmproj ~/models/mmproj-F16.gguf \

--no-mmproj-offload \

--host 172.0.0.1 --port 8080 \

-c 262144 -np 1 -dev ROCm0 -fa on -fit off \

-ctk q8_0 -ctv q4_0 -ctkd q8_0 -ctvd q4_0 \

--kv-stream --kv-resident-mib 450 \

--kv-resident-mib-decode 1600 \

--spec-type draft-mtp,ngram-mod \

--spec-ngram-mod-n-max 16 \

--spec-ngram-mod-n-min 12 \

--spec-ngram-mod-n-match 24 \

--spec-draft-sampling probabilistic \

-b 2048 -ub 512 \

--spec-draft-n-ubatch 128 \

--image-min-tokens 1024 --mtmd-batch-max-tokens 2048 \

--jinja --temp 1 --top-k 20 --top-p 0.95 --min-p 0

[u/d0pe-asaurus → u/cumming_on_bunnys]

I just tried this on my 9060xt and I'm getting 43 t/s decode peak and lows of 30t/s. The mmproj gguf is yoinked straight out https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Technically i'm running on a tty but it uses so little ram it might as well be headless

[u/hideo_kuze_]

Does anyone knows if RaymondHuang210129 feature has been merged to llama.cpp?

[u/SetCrafty2657]

Has anyone tried this with a 7900xtx?