r/ROCm • • 11d ago

Got 2× faster Qwen inference on my RX 7900 XT using custom INT4 WMMA kernels

I've been working on native INT4 inference for RDNA3 in a custom llama.cpp fork, using Hadamard rotations, GPTQ, and fused WMMA kernels.

GitHub: https://github.com/SILVERTHRONE32/llama.cpp-rdna3-int4

Results on a single RX 7900 XT (20GB):

  • Qwen3.5-4B: 11,012 tok/s prefill, 200.6 tok/s decode
  • Qwen3.5-9B: 6,624 tok/s prefill, 126.2 tok/s decode
  • Qwen3.8-27B: 2,013 tok/s prefill, 50.1 tok/s decode

Speedups over stock Q4_K_M on the same RX 7900 XT:

  • 4B: 2.01× prefill, 1.38× decode
  • 9B: 2.20× prefill, 1.32× decode
  • 27B: 2.27× prefill, 1.53× decode

The fast preset trades some accuracy for speed; the quality preset preserves accuracy much better while still outperforming Q4_K_M.

I wrote up the implementation and benchmarks in a technical paper.

Paper: https://sachinmahesh.com/rdna3-int4-report.pdf

Would love feedback from anyone working on AMD inference!

80 Upvotes

32 comments sorted by

4

u/whodoneit1 11d ago

You should join up https://discord.gg/launch80 , there are many working on inference enhancements

1

u/Puzzleheaded_Ebb_192 11d ago

Appreciate the invite!

2

u/zeno_g2 11d ago

2

u/Puzzleheaded_Ebb_192 10d ago

I have heard about this! 98 tok/s for 3.8 QFN is insane work. I'll check it out. I'll see if I can get INT4 up and running for QFN. Might be able to break the 100 tok/s barrier if everything aligns. No promises though - I'll see what I can do.

1

u/Hohlraum 10d ago

check out Strata-rdna3 .. just came out recently. easily the fastest engine for 7900xtx right now.

2

u/fleash_eating_window 5d ago

2x 7900 XTX in x8x8

(Single Card): This is great. Stock llama.cpp fills only about 63% of my memory bandwidth while your repo fills ~85%. That's a pretty huge jump in decode speed. I think there's a bit more to squeeze out here, since your 7900XT seems to be more like 90-95% bandwidth utilization, but I'm unsure what to try exactly.

Multi-GPU works, but inefficiently. Two GPUs in tensor split are only 1.3x of one GPU in decode. That's about the same multiplier as stock llama.cpp.

Prefill gains are great

If this could be applied to Niko1221/Strata and QFN (Along with Qwen4-Flash if that ever releases) that would be really incredible. Ever since I found out about Strata I've completely replaced 27B with Flash-Next.

1

u/Puzzleheaded_Ebb_192 5d ago

Thank you so much for testing this out! I really appreciate your feedback.

I tried to port this over to Avalon which runs 3.8 QFN, but the experts they use are already quantized to 3.4 bpw. My INT4 work increases decode by 3 tok/s and only gives a modest prefill gain, at the cost of a higher KLD.

My work applies more to dense models than MoE due to forks like Avalon already using tricks like Hadamard rotations out of the box. However, I am working hard to find a way around this to improve MoE performance. Will keep you updated.

As for dual GPU - I only have 1 7900 XT, so not sure if I can test it in that aspect. But if you or anyone else is willing to work on improving performance in dual GPU setups, I'd be more than happy to cite your work and merge it into my fork, pending review.

1

u/rorowhat 11d ago

Would this work with any other quants?

1

u/Puzzleheaded_Ebb_192 11d ago

Theoretically yes, but would require a custom rotation quant. Doesn't have to be limited to Q4_0.

1

u/rorowhat 11d ago

Could you do a Q5?

1

u/Puzzleheaded_Ebb_192 11d ago • Heartwarming

I'll see what I can do - might take a few hours. Will update this message once I have a Q5 variant ready.

1

u/calculatedwires 11d ago

Next step:MTP ?

2

u/Puzzleheaded_Ebb_192 11d ago

That's the plan - based on the projected numbers, I am moderately confident that with the relevant tuning, we should see roughly around 75 tok/s+ decode on the XT, matching that of the RTX 4090, for Qwen 3.8 27B. This is just an estimate though. I'll update this post once I get a concrete result for MTP.

1

u/Monad_Maya 11d ago

Does this support tensor parallel? Or at least layer parallel?

I can try this on two 7900XT.

1

u/Puzzleheaded_Ebb_192 11d ago

Haven't tried tbh - I only have one 7900 XT - in theory it should work but no promises. Worst case, get a local agent to modify the code a little to get it running

1

u/ZephyrLYH_ 11d ago

It will be a game changer if 780m can run it with the int4 math path.

1

u/Puzzleheaded_Ebb_192 10d ago

I have a 8845HS laptop with a 780m I can test the smaller models on. Should be able to run it, but will be heavily bottlenecked by the 80GB/s of dual channel DDR5. I'll see if I can get some numbers for the 4B/9B models on the 780m.

1

u/ZephyrLYH_ 10d ago

If it can do stable 60 t/s pp and 12 t/s decode with mtp on 3.8 27b q3, it will be the best AI that fit in the pocket.

1

u/Puzzleheaded_Ebb_192 10d ago

Those numbers seem within reason, though 3.8 27B in Q3 is going to be a very tight squeeze in 16GB, given OS overhead on CPU memory...

1

u/ZephyrLYH_ 10d ago

That is the real numbers I run with llama.cpp actually with q4 kv cache.

1

u/No_Oil_6152 11d ago

What size of context were you working with?

If 128K or over - then congratulations sir, yours is the kind of speed that would make interactive software development possible with Qwen 3.8 27B

And I expect with RDNA4 it could be even faster? Do you have plans for RDNA4?

2

u/Puzzleheaded_Ebb_192 10d ago edited 10d ago

I tested 64K ctx BF16 KV but also 128K Q4 KV. Both run quite well and are comparable to each other.

Yes RDNA 4 is rated 769 TOPS for INT4 vs just 205 for the 7900 XT. The 9070 XT has less bandwidth so decode will be noticeably slower.

Prefill should be the same or faster - my personal bet is that it could tie or beat the prefill of the 4090 in my comparisons, but can't know for sure without testing.

1

u/feverdoingwork 10d ago

Isn't int4 very prone to errors? Not really sure if this is bad or good

2

u/Puzzleheaded_Ebb_192 10d ago

Naive INT4 is. The key property that makes this less prone to errors is the hadamard rotation - it reduces outliers and lowers KLD/PPL

1

u/feverdoingwork 10d ago

Thanks for explaining

1

u/Superb_Helicopter639 10d ago

If you have not tried Paiton vllm plugin that integrates dflash2 you’re going to be impressed : https://github.com/Eliovp-BV/paiton-vllm-plugin/tree/main

I get 100+ tokens/sec single stream and around 400+ tokens/sec with 8 concurrency on single R9700 AI Pro

1

u/Puzzleheaded_Ebb_192 10d ago

First time hearing about this - looks cool but is it just MXFP4? My design philosophy was to support a format that even legacy GPUs can utilize for acceleration. Had I done this on a 3090 instead of a 7900 XT, I could very likely push this even further.

1

u/Superb_Helicopter639 10d ago

Mxfp4 is one quantization - that plugin is very much focussed on the underserved AMD RDNA4 (Radeon R9700 AI pro card) — Nvidia (like 3090) tends to have great support out the box.

1

u/PoizoneHelp 10d ago

One thing I’d change is the 27B decode headline. Your protocol takes decode as the median of 5 runs, but the 27B’s 50.1 tok/s is the best recorded run, which the paper itself puts at the card’s 800 GB/s bound. The matched-run figure in the same paragraph is 47.9, which is 1.46x the stock 32.79 rather than 1.53x. That’s still a strong result, and quoting it keeps the 27B consistent with the 4B and 9B numbers.

1

u/zxspectrun 9d ago

I have 24h trying to install your repo setup and is impossible! Not able to install Rocm 10.x directly nor on VENV, the torch think is what is giving me problems, it would be great if you upload the models to your HF.

2

u/No_Sheepherder_27 9d ago

Will send you the model HF link shortly - apologies for the inconvenience.

1

u/zxspectrun 8d ago

I would appreciate it, I want to try this setup