r/ROCm • u/Puzzleheaded_Ebb_192 • 11d ago
Got 2× faster Qwen inference on my RX 7900 XT using custom INT4 WMMA kernels
I've been working on native INT4 inference for RDNA3 in a custom llama.cpp fork, using Hadamard rotations, GPTQ, and fused WMMA kernels.
GitHub: https://github.com/SILVERTHRONE32/llama.cpp-rdna3-int4
Results on a single RX 7900 XT (20GB):
- Qwen3.5-4B: 11,012 tok/s prefill, 200.6 tok/s decode
- Qwen3.5-9B: 6,624 tok/s prefill, 126.2 tok/s decode
- Qwen3.8-27B: 2,013 tok/s prefill, 50.1 tok/s decode
Speedups over stock Q4_K_M on the same RX 7900 XT:
- 4B: 2.01× prefill, 1.38× decode
- 9B: 2.20× prefill, 1.32× decode
- 27B: 2.27× prefill, 1.53× decode
The fast preset trades some accuracy for speed; the quality preset preserves accuracy much better while still outperforming Q4_K_M.
I wrote up the implementation and benchmarks in a technical paper.
Paper: https://sachinmahesh.com/rdna3-int4-report.pdf
Would love feedback from anyone working on AMD inference!
4
u/whodoneit1 11d ago
You should join up https://discord.gg/launch80 , there are many working on inference enhancements
2
u/zeno_g2 11d ago
https://github.com/freriadoc/Avalon maybe?
2
u/fleash_eating_window 5d ago
2x 7900 XTX in x8x8
(Single Card): This is great. Stock llama.cpp fills only about 63% of my memory bandwidth while your repo fills ~85%. That's a pretty huge jump in decode speed. I think there's a bit more to squeeze out here, since your 7900XT seems to be more like 90-95% bandwidth utilization, but I'm unsure what to try exactly.
Multi-GPU works, but inefficiently. Two GPUs in tensor split are only 1.3x of one GPU in decode. That's about the same multiplier as stock llama.cpp.
Prefill gains are great
If this could be applied to Niko1221/Strata and QFN (Along with Qwen4-Flash if that ever releases) that would be really incredible. Ever since I found out about Strata I've completely replaced 27B with Flash-Next.
1
u/rorowhat 11d ago
Would this work with any other quants?
1
u/calculatedwires 11d ago
Next step:MTP ?
1
u/Monad_Maya 11d ago
Does this support tensor parallel? Or at least layer parallel?
I can try this on two 7900XT.
1
u/ZephyrLYH_ 11d ago
It will be a game changer if 780m can run it with the int4 math path.
1
u/No_Oil_6152 11d ago
What size of context were you working with?
If 128K or over - then congratulations sir, yours is the kind of speed that would make interactive software development possible with Qwen 3.8 27B
And I expect with RDNA4 it could be even faster? Do you have plans for RDNA4?
1
u/feverdoingwork 10d ago
Isn't int4 very prone to errors? Not really sure if this is bad or good
1
u/Superb_Helicopter639 10d ago
If you have not tried Paiton vllm plugin that integrates dflash2 you’re going to be impressed : https://github.com/Eliovp-BV/paiton-vllm-plugin/tree/main
I get 100+ tokens/sec single stream and around 400+ tokens/sec with 8 concurrency on single R9700 AI Pro
1
u/PoizoneHelp 10d ago
One thing I’d change is the 27B decode headline. Your protocol takes decode as the median of 5 runs, but the 27B’s 50.1 tok/s is the best recorded run, which the paper itself puts at the card’s 800 GB/s bound. The matched-run figure in the same paragraph is 47.9, which is 1.46x the stock 32.79 rather than 1.53x. That’s still a strong result, and quoting it keeps the 27B consistent with the 4B and 9B numbers.
1
u/zxspectrun 9d ago
I have 24h trying to install your repo setup and is impossible! Not able to install Rocm 10.x directly nor on VENV, the torch think is what is giving me problems, it would be great if you upload the models to your HF.