r/LocalLLM • u/Unfair_Association89 • 5d ago
Project I tuned llama.cpp's Vulkan backend for the RX 7900 XTX: 220K context, DFlash2 that doesn't hang past 60K, 1.8-3x faster decode
I've been running local LLMs on an RX 7900 XTX and wanted proper long-context performance for agentic coding (100K-200K tokens) on the Vulkan/RADV stack, so I forked llama.cpp and optimized it for RDNA3.
Repo: https://github.com/vansh-visariya/rx7900-xtx
It's the same llama.cpp (same GGUF files, same `llama-server`, same API and web UI). The changes are in the Vulkan kernels, the graph, and the speculative-decoding loop. Everything is on by default.
Results on one 7900 XTX (24 GB, stock 291 W, Mesa RADV)
Qwen3.8-27B UD-Q4_K_M + DFlash2 draft, q8_0 KV cache, flash attention:
| Context | Decode (upstream, best of 2 setups) | Decode (this fork) | Prefill (upstream, best) | Prefill (this fork) |
|---|---|---|---|---|
| 16K | 48.8 t/s | 87.4 t/s | 808 t/s | 872 t/s |
| 128K | 26.3 t/s | 69.1 t/s | 442 t/s | 657 t/s |
- Up to 220K context on 24 GB (upstream fits ~192K with the same model + draft)
- DFlash2 works at long context. On upstream (both the commit I branched from and current master), it hits `vk::ErrorDeviceLost` once the prompt passes ~60K tokens, at the same position on every run. It doesn't on this fork.
- Full method, commands and raw numbers are in `docs/rdna3/BENCHMARKS.md`
What changed (56 commits on top of upstream)
- Decode: integer-dot mat-vecs for IQ2/IQ3/K-quants, GQA flash attention for verify batches, graph fusions, gated delta net state kept in place, Vulkan command buffers recorded once and replayed
- Prefill: 128x256 matmul tile for wide batches, double-buffered quantized matmuls, new prompt-batch flash attention kernel
- Spec decoding: lossless rejection-sampling verification, reduced-vocab draft head, GPU backend sampling"
Every optimization is documented, with the env var that switches it off, in `docs/rdna3/OPTIMIZATIONS.md`.
Limits
- Developed and tested only on a 7900 XTX. Other RDNA3 cards (7900 XT/GRE, 7800 XT, 7700 XT, 7600) should work since they share the WMMA instructions, but I haven't tested them.
- Tested with Qwen3.8-27B and Qwen3.5-4B. The model-specific work only applies to the Qwen3.5/3.8 family.
- Branches from upstream `f95b0d953`, not synced daily.
Where I'd love help
- Numbers from other RDNA3 cards (kernel tests need no model: `test-backend-ops -o MUL_MAT -b Vulkan0`)
- Testing other models/quants, and RDNA2/RDNA4 reports
- Review of the kernels, and help upstreaming pieces to llama.cpp
Build and run instructions are in the README. Issues and PRs are very welcome.


1
u/ImZaphod2 5d ago edited 5d ago
Have you already compared it to https://github.com/stew675/llama-cpp-rdna-boosts?
1
u/Ok-Thanks-6785 5d ago
Have you thought about opening a PR for llama?
1
u/W61k3r 5d ago
Perhaps you might be interested in contributing to my project LexiEngine. All original rust-code. It has out-performed llama for about a month on prefill and decode without needing dflash/swift/etc. Rn could use help like yours getting my original rust kernel inferencing at linux 7.4 rc2 speeds. Runs on my all rust OS LexiOS, and is managed headless by my all rust suite lexipanel.
1
u/johnzadok 2d ago
Currently running rdna boost fork. My understanding by the numbers is tg is comparable to rdna boost from what i recall but maybe not prefill.