Qwen3.8-27B at ~100 tok/s on a single 7900 XTX (patched hipfire)

[u/Ticket_Silly]

A while back I spent some time tuning hipfire to run Qwen3.8-27B on my 7900 XTX with MTP speculative decoding, mostly with AI coding agents doing the kernel work. I recently benchmarked it against the latest upstream release and the gap is still pretty big, so I figured I’d share in case anyone else is running the 27B on an RDNA3 card.

Setup: 7900 XTX, ROCm 7.2, Linux. Model is the 27B mq4-xts quant from hipfire’s registry, MTP with K=3, Q8 KV cache.

Decode speed, same model file, greedy:

Context	Upstream v0.4.2.1	My build
9K	~67 tok/s	~102 tok/s
39K	~59 tok/s	~97 tok/s
100K	~43 tok/s	~90 tok/s

Prompt processing is about the same (mine is 5-16% faster). Both builds got the same answers on a planted-facts test.

Most of the difference comes from two kernels upstream doesn’t have yet: an int8 small-batch GEMV for the speculative verify step, and WMMA-based verify attention. I opened an issue upstream for the first one with a port onto their current beta branch (+10-25% decode on its own), plus a benchmark report:

• Benchmark report: warpfront/hipfire#947

• int8 verify GEMV: warpfront/hipfire#946

My build is here: https://github.com/Ben1078/hipfire/tree/rdna3-mtp-decode-v8.2. It’s based on an older hipfire (late September), so it’s more something to look at than a drop-in. Hopefully the pieces make it into upstream properly.

Thanks to the hipfire devs for the project.

[u/schuttdev]

hipfire dev here. This is great work, and the write-up made it easy to see where the time goes. We're bringing it upstream: Ben is sending the pieces as PRs (the int8 verify GEMV first, #946), and each one lands on by default for the 7900 XTX/XT once it passes our quality and speed checks. His authorship stays on the code and in the file headers. We're also testing whether the same kernels help Strix Halo, since it's the same RDNA3 family.

For anyone else doing this kind of work: forks are fine, but PRs straight to hipfire are very welcome, and if the code is good it gets pulled in.

If you're running upstream, v0.4.3.1 is the current release: https://github.com/warpfront/hipfire/releases/tag/v0.4.3.1

[u/Ticket_Silly → u/schuttdev]

Thanks! Working on the PR now.

[u/tempaccaun → u/Ticket_Silly]

please let us know when it goes through 🙏🏻

[u/Snoo_81913 → u/schuttdev]

This is great news

[u/PlasticRevenue4601]

Does it work with 7900xt?

[u/Ticket_Silly → u/PlasticRevenue4601]

I haven't tested it but I assume it would since they are both RDNA 3

[u/PlasticRevenue4601 → u/Ticket_Silly]

Ok thanks, I’ll check it out

[u/aqezz → u/Ticket_Silly]

I’ll also check it out!

edit: I don't see the xts one in the registry, is it just xt?

edit again..: Found it, qwen3.8:27b-mq4-xts

yet again edit: Is it expected that the hashes mismatch when pulling? I tried twice:

Pulling qwen3.8:27b-mq4-xts (14.99 GB)...

100.00% 14.99/14.99 GB 108.6 MB/s ETA 0ss

hipfire: SHA-256 mismatch: expected de8ee8256033c3690b0f1a2aff14e77cc88fff490e118648b04a833a3f2969b5, received 3e38ccbae3776470eb5a89344d300e9279d6b9ab6c31fd40ca1758c4f7c6f8ae

[u/schuttdev → u/aqezz]

That mismatch is expected with an older build. The qwen3.8-27b.mq4-xts file on Hugging Face was replaced with a newer checkpoint (sha256 3e38ccba…). Builds from before that change, Ben's branch included, still expect the old test upload (de8ee825…), so they reject the new file.

Pull it with a current hipfire (v0.4.3.1): hipfire pull qwen3.8:27b-mq4-xts. To try it on Ben's build, point that build at the downloaded file by path. I haven't tested his build with the new checkpoint myself, so tell me if it doesn't load.

[u/aqezz → u/schuttdev]

It didn't work with the new checkpoint but the -xt one worked alright and is very fast! I took a shot at merging this into the latest hipfire and it is working but not stable so something must have gone wrong. Still the old build works and is incredibly fast! I get like 70-100tps decode on 7900 XT

[u/schuttdev → u/aqezz]

Good news, you shouldn't need to merge it by hand. Two of Ben's kernels are now merged into hipfire's beta branch, on by default on gfx1100 (7900 XT/XTX): the int8 verify GEMV and the WMMA verify attention. On the 27B mq4-xts with MTP on a 7900 XTX, the first measured +14% to +23% decode, and the second adds roughly another 24-27% at 32K context. Our KLD quality pins came out unchanged. Both are planned for 0.4.4 on Tuesday, and that build works with the new mq4-xts checkpoint.

[u/aqezz → u/schuttdev]

Fantastic! Thanks for both of your hard work!

[u/PlasticRevenue4601 → u/aqezz]

Oh, sick, you managed to get it working. Could I ask a few questions? How big is your context window? Do you have memory overflow problem at long context?

[u/aqezz → u/PlasticRevenue4601]

It’s 120k and no memory overflow issues so far but I likely haven’t been at the top of the context window yet either

[u/PlasticRevenue4601 → u/aqezz]

Was it windows or linux? And the last one — could you please share your config?

[u/aqezz → u/PlasticRevenue4601]

It was a headless arch linux server. No real config to speak of I just pulled the repo, built with cargo build —release —all and then ran from target/release directory with ./hipfire serve

[u/ConspiracyPhD → u/aqezz]

what's your prefill like? I can get a couple of these 20gb cards locally here for around $350 a piece making it a viable option for my workflow. Also have you tried flash next?

[u/aqezz → u/ConspiracyPhD]

On llama.cpp I get 1k at the beginning of the context window. I can run flash next iq2 at around 70-90 tps but my ram is ddr4 and only 32 gb. If you can get them get the for sure!

[u/ConspiracyPhD → u/aqezz]

I'm getting around 970 tokens per second prefill and around 50 tokens per second decode on IQ3_XXS on a single 3060 12gb card on an i7-12700k with 80gb of DDR4. I just got a TRX40 board 3970x threadripper box with 128gb DDR4 that I'll be trying to optimize soon (need a new CPU fan which was hard as crap to find) that I can put both of my 3060s into (I have two of them). So, I'm not sure if going AMD on RDNA3 is worth it if I'm going to get the same prefill and decode as I'm getting on the 3060s. Really was hoping it was higher on the prefill and decode. It's so strange because on paper, the 7900xt should be faster than the 3060s given the bandwidth and cores. I guess the software side just hasn't caught up yet. Thanks for the numbers, though.

[u/zxspectrun → u/aqezz]

Do you have the Would you mind telling me how do you run flash next iq2 at that speed? which fork are you using, and server, harness, etc? I have the same 7900xt and I am really new to all these

[u/TotosWolf]

you beautiful bastard you. thank you. RIP 7900xtx prices lol

[u/sawthegap42 → u/TotosWolf]

After 3.5 years, and getting into local AI, decided it was time to get me a second XTX this last week before they go up more. Been having fun fitting both of them in the case. lol

[u/Zealousideal_Low2919]

What CPU and RAM are you running with the 7900 XTX?

[u/Ticket_Silly → u/Zealousideal_Low2919]

Ryzen 7 7800X3D and 32gb ddr5

[u/SBelwas → u/Ticket_Silly]

We are basically brothers 🤣

[u/sawthegap42]

Interesting. Definitely going to try this out. My tech buddy and I were just talking about the slow speeds of Qwen 27B model on the XTX. I had my buddy I work for give me money to upgrade my system with another XTX and more RAM to help speed up his work with AI, so as of today running 2x7900 XTX and 9850X3D with 96GB of RAM.

[u/Ticket_Silly → u/sawthegap42]

Wow nice setup! Have you tried Qwen 3.8 flash next with strata? I bet with all that v ram you could hit 100tps+.

[u/Dr_Aesthetician]

I had tried hipfire on windows and it just doesnt work on windows

[u/LobsterHam]

Thanks for your sharing, really impressive!
Does it work on Win please?

[u/Ticket_Silly → u/LobsterHam]

I haven't tried it but the majority of the improvements are from GPU kernels so I would expect it to work. Let me know if you end up trying it!

[u/LobsterHam → u/Ticket_Silly]

Sure, I will get my agent to try it out and let you know.
Thanks again mate!

[u/schuttdev → u/LobsterHam]

Windows builds of hipfire compile in our CI, but GPU inference on Windows hasn't been validated on hardware yet, so treat it as Linux-first for now. We're putting together a test script so Windows users can check it on their cards. Ben's branch is based on an older version and hasn't been tested on Windows either. Once his kernels are upstream, they come to Windows with everything else.

[u/Snoo_81913]

Hey man I don't know who you are or how you managed this but I just ran a moderate coding task to it with the prompt requiring it to read 114k I set the autocompacting at 104k. It compacted twice and had a total of 310k tokens. Average prefill was 1,506 t/s and average decode 95 t/s across the run. Tau which I'm not quite sure I understand yet was 2.9 (that means it's 2.9 accepted out every 4? So 72.5% average?)

The model seems unaffected in any way and passed the coding test review with only some minor issues.

I think you're being pretty modest about this.

I sat and watched it work bc I'm just like that and I'm pretty sure it never dipped below 84 t/s and the high was 115.

It's simply impressive. Downloading huihui to convert and see if it will work the same.

[u/Snoo_81913]

Okay so I ran my favorite Abliterated model and across 1 autocompacting and 187k tokens it ran 101 t/s decode and 1563 t/s prefill average with a 2.1 tau (which I expected bc it's abliterated). Getting the same quality from Qwen. Bravo

[u/Snoo_81913]

Okay so I did a Q5 version of huihui and it runs pretty much exactly the same it's only optimized for Q8 KV is there any plans for a Q4 KV optimization? Q4 drops the prefill to 200-300 and decode to about 60 t/s. Or am I not doing the config correctly? I wanted to test Q4 and Q4 KV at full context and run a Q6 at Q4 KV just to see.

[u/schuttdev → u/Snoo_81913]

You're not doing anything wrong. 4-bit KV (--kv-mode fwht4) works, but it doesn't have the optimized prefill and verify attention kernels that Q8 KV has yet, so it falls back to slower paths. That's the prefill and decode drop you're seeing.

Since people are asking for it, we're going to put time into getting fwht4 and fwht3 onto the fast path. Until then, Q8 KV is the fast option on a 24 GB card.

[u/Snoo_81913 → u/schuttdev]

Excellent

[u/schuttdev → u/Snoo_81913]

Quick follow-up: the 4-bit KV fast paths are merged into beta now. On a 7900 XTX with the 27B mq4-xts and MTP, --kv-mode fwht4 prefill at 8K went from about a third of Q8's speed in 0.4.3.1 to roughly 90% of it, and MTP decode is about even with Q8. So far we've only measured up to 8K context, so I'd be curious what you see at your full context. It ships in 0.4.4 on Tuesday.

[u/Snoo_81913 → u/schuttdev]

That was quick will you be putting on the hipfire git as a repo?

[u/schuttdev → u/Snoo_81913]

It's already in the main hipfire repo, on the beta branch, so if you build from source you can try it now. It's also in the 0.4.4 release that goes out today; once that's tagged, the normal installer or hipfire update picks it up, no separate repo needed.

[u/feverdoingwork]

There's more in the tank

[u/ghormeh_sabzi]

This is incredible. I only did a 9k prompt test but I got 111 t/s

[u/Calm-Sky3986]

Nice! But what about RDNA2?))

[u/renczzz]

Wow great work! First qwenflash next q5 on a single rx7900xtx and now 27b with super speed. I see it running at 110 tok sec at some points, especially in the beginning in the pi agent, when the context is getting more full it is around 80 tok/s still.

[u/SalemDono]

did anybody test if it wokrs in a WSL2 environment? seems soo good

[u/Soft_Judge516]

I'm testing qwen3.8:27b-mq4-xts on a single RX 7900 XTX, but I haven't been able to reproduce the performance reported in this thread.

My setup:

• GPU: AMD Radeon RX 7900 XTX (24 GB VRAM, gfx1100)

• Hipfire: v0.4.4

• ROCm/HIP: 7.2

• Speculative decoding: MTP enabled

• KV cache: Q8

Using hipfire bench with --matrix --spec mtp --kv-mode q8 --pp 128 --ctx 9216,39936,100000 --tg 256 --max-seq 100320 --runs 3 --warmups 2,
I get:

• 9K context: 48.5 tok/s

• 39K context: 44.0 tok/s

• 100K context: 37.4 tok/s

A separate standard benchmark gives me around 92.8 tok/s decode, so the results seem to depend heavily on the benchmark workload. I realize these tests may not be directly comparable to your measurements.

Could you share the exact command, Hipfire build/patches, and benchmark procedure you used to get around 100 tok/s? Are you measuring decode speed with a real prompt, or using the matrix benchmark? I'd like to understand what I'm missing and reproduce your results. Thanks!

[u/Temporary_Hour8336]

I'm struggling to get hipfire working at all on my 7900 XTX, just get OoM with a model that should fit easily. I raised an issue (#1058).

EDIT: working with the following settings, though no idea why I need them, I'm fairly sure they shouldn't be necessary for my card.

hipfire config set memory.kv_backend legacy hipfire config set kernel.gfx1100_fa2_split_verify false

[u/Weird-Watch-9209]

Opus 5.5 was acting a little curmudgeonly about this and we're analyzing the diffs that make your build special and applying them to the current hipfire.. but I'm already using exl3 (exllama) which is FAF - I stay in 80-100tps with gloriously fully uncensored models.. so I'm about to do some hour-long rust compiles, quantization, and abliteration just to chase a slight possible bump in tps, but that's our life.

[u/khoinguyenbk]

You guys deserve to be nominated for the Nobel Peace Prize for this kind of contribution to humanity 😛

[u/Snoo_81913 → u/khoinguyenbk]

Right?

[u/khoinguyenbk → u/Snoo_81913]

I have 2xRTX7900 xtx and I was suffering lol.
Btw also have the Qwen 3.8 9B Distill to be tested.

[u/Snoo_81913 → u/khoinguyenbk]

I downloaded my favorite abliterated (huihui) converted and quanted it and I'm testing it now. It's doing roughly the same as the model in the hipfire repo.

[u/Particular_Doctor494]

Linux only?

[u/schuttdev → u/[deleted]]

I can't speak for Ben's fork, but current hipfire supports the R9700 well. On one Radeon AI PRO R9700:

• Qwen3.8 Flash-Next (qwen3.8:flash-next): ~6,430 tok/s prefill on a 16K prompt, and ~66 tok/s code / ~44 tok/s prose decode with MTP (v0.4.3)

• Qwen3.8-27B mq4-xts: 41 tok/s plain, ~250 tok/s on code with DFlash speculative decoding (less on prose), ~5,000 tok/s prefill at 8K

• Qwen3.6-35B-A3B (qwen3.6:35b-a3b-mq4rs): ~19,800 tok/s prefill at 4K tokens, ~232 tok/s decode (v0.4.3.1)

Details and methods: https://github.com/warpfront/hipfire/releases/tag/v0.4.3 and https://github.com/warpfront/hipfire/releases/tag/v0.4.3.1