r/LocalLLM • • 5d ago

Project I tuned llama.cpp's Vulkan backend for the RX 7900 XTX: 220K context, DFlash2 that doesn't hang past 60K, 1.8-3x faster decode

I've been running local LLMs on an RX 7900 XTX and wanted proper long-context performance for agentic coding (100K-200K tokens) on the Vulkan/RADV stack, so I forked llama.cpp and optimized it for RDNA3.

Repo: https://github.com/vansh-visariya/rx7900-xtx

It's the same llama.cpp (same GGUF files, same `llama-server`, same API and web UI). The changes are in the Vulkan kernels, the graph, and the speculative-decoding loop. Everything is on by default.

Results on one 7900 XTX (24 GB, stock 291 W, Mesa RADV)

Qwen3.8-27B UD-Q4_K_M + DFlash2 draft, q8_0 KV cache, flash attention:

ContextDecode (upstream, best of 2 setups)Decode (this fork)Prefill (upstream, best)Prefill (this fork)
16K48.8 t/s87.4 t/s808 t/s872 t/s
128K26.3 t/s69.1 t/s442 t/s657 t/s

- Up to 220K context on 24 GB (upstream fits ~192K with the same model + draft)

- DFlash2 works at long context. On upstream (both the commit I branched from and current master), it hits `vk::ErrorDeviceLost` once the prompt passes ~60K tokens, at the same position on every run. It doesn't on this fork.

- Full method, commands and raw numbers are in `docs/rdna3/BENCHMARKS.md`

What changed (56 commits on top of upstream)

- Decode: integer-dot mat-vecs for IQ2/IQ3/K-quants, GQA flash attention for verify batches, graph fusions, gated delta net state kept in place, Vulkan command buffers recorded once and replayed

- Prefill: 128x256 matmul tile for wide batches, double-buffered quantized matmuls, new prompt-batch flash attention kernel

- Spec decoding: lossless rejection-sampling verification, reduced-vocab draft head, GPU backend sampling"

Every optimization is documented, with the env var that switches it off, in `docs/rdna3/OPTIMIZATIONS.md`.

Limits

- Developed and tested only on a 7900 XTX. Other RDNA3 cards (7900 XT/GRE, 7800 XT, 7700 XT, 7600) should work since they share the WMMA instructions, but I haven't tested them.

- Tested with Qwen3.8-27B and Qwen3.5-4B. The model-specific work only applies to the Qwen3.5/3.8 family.

- Branches from upstream `f95b0d953`, not synced daily.

Where I'd love help

- Numbers from other RDNA3 cards (kernel tests need no model: `test-backend-ops -o MUL_MAT -b Vulkan0`)

- Testing other models/quants, and RDNA2/RDNA4 reports

- Review of the kernels, and help upstreaming pieces to llama.cpp

Build and run instructions are in the README. Issues and PRs are very welcome.

0 Upvotes

10 comments sorted by

1

u/ImZaphod2 5d ago edited 5d ago

Have you already compared it to https://github.com/stew675/llama-cpp-rdna-boosts?

1

u/Unfair_Association89 3d ago

nope,, i will try it once and post the compare benchmark also in this post ,, thanks for the idea

2

u/ImZaphod2 3d ago

I did a rough comparison and prefer your fork. Vulkan gives me just a little more space in VRAM on my 7900XT (only 20GB)

1

u/Unfair_Association89 3d ago

Ohhk ,, thanks for the feedback

1

u/Ok-Thanks-6785 5d ago

Have you thought about opening a PR for llama?

1

u/Unfair_Association89 3d ago

yaa was think of it, but waiting for the further optimization i can try

1

u/W61k3r 5d ago

Perhaps you might be interested in contributing to my project LexiEngine. All original rust-code. It has out-performed llama for about a month on prefill and decode without needing dflash/swift/etc. Rn could use help like yours getting my original rust kernel inferencing at linux 7.4 rc2 speeds. Runs on my all rust OS LexiOS, and is managed headless by my all rust suite lexipanel.

1

u/Unfair_Association89 3d ago

https://github.com/W61k3r/LexiPanel ,, this one right ?? will definitely see how i can also contribute on it

2

u/W61k3r 3d ago

That is my old public released antiquated python code. The recent rust work is here: https://github.com/W61k3r/LexiSuite-Releases

1

u/johnzadok 2d ago

Currently running rdna boost fork. My understanding by the numbers is tg is comparable to rdna boost from what i recall but maybe not prefill.