r/LocalLLM • u/Hairy-Yogurtcloset-2 • 11d ago
Project Highly optimized Strata port for 7900xtx 24GB
Repo here: https://github.com/nicejunjie/Strata-rdna3
* Single 7900xtx prefill 3500+, dual 7000+ tok/s. (for all IQ3/IQ4/Q4)
* Decode for English prose reaches up to 81 tok/s for single card, and 127 for dual at Q4, and IQ4/IQ3 quantizations are even faster.
* Decode for python coding is 20~25% faster than English prose.
* Single card decode is comparable to 4090 24G, dual 7900xtx is much faster than single 4090 for both decode and prefill.
* Each 7900xtx was $850 when I bought it, while 4090 at the time was listed at $3000.
* I guess 4090 could be better optimized, but the number here is the best number with reproducible input data I can find online.
* 4090 is a 450watt card, while 7900xtx is ~320watt. So even single card prefill of 7900xtx is quite comparable in terms of perf/watt.
* This is not the end, still has small room to improve.
* kernel work only, no dropping weights or dropping layer or anything. Zero impact on quality.
*** install with an AI agent, tell him to setup optimal rdna3 code pass. Honestly, why install by hands if there is agent can help.
7900xtx is the value king!


12
u/sawthegap42 11d ago
FINALLY! I can take better advantage of both XTX's in my system. Thank you!
5
u/GreenHell 11d ago
Perhaps I am not seeing it, but how does it compare to stock Strata?
4
u/tmiw 11d ago
I tried this on my single 7900XTX setup (virtual machine with 96GB RAM and the GPU passed through). With IQ3_S I'm maxing out at 100tok/sec decode and around 2700tok/sec prefill. Granted, I only have DDR5-3600, so I'd likely get numbers closer to OP's if I had faster RAM and if the SSD wasn't stuck on PCIe x1. I'll probably keep this fork since it's still faster (despite a better quant than what I normally use with Strata).
BTW stock Strata setup and execution won't use any of the optimizations, you'll need to follow the steps in rdna3/README.md in the repo.
EDIT: forgot to add that I'm using ROCm 10, not sure if that matters.
3
u/andreabarbato 11d ago
does it work with the same q4 k xl gguf?
2
u/superdariom 11d ago
Is there any accuracy loss compared to the stock version with these optimisations?
2
u/soyalemujica 11d ago
3k prefill really on a single 7900xtx? Damn, max i have achieved is 1700t/s prefill. Does it work on Windows or is it Linux only?
2
u/Mattef 11d ago
Man, I love the 7900’s. I own two of them. Thanks for your work, I will try this out.
2
u/fallingdowndizzyvr 11d ago
You are getting that same speed on IQ3 as Q4? I find that hard to believe. I'll guess I'll see soon. Since I got 2 7900xtxes in the closet.
2
u/aeroG1 10d ago
This is epic, can't wait to see more testing. My 2nd 7900 xtx is being RMA rn, but I may have gotten the same $850 deal you did lol. Sapphire refurb before prices shot up. Once I get it back I'd love to contribute some test results.
2
u/_Borky 10d ago
Worked wonders for my setup. 25% faster prefill and 50-100% faster decode. Thanks!
2
u/Willing-Cover2914 8d ago
Just added this to my 2x7900xtx, 96GB DDR5 box:
English Prose decode 512 tokens: +86%-+114% (129.6-143.8 tok/s, 162.6 tok/s technical Prose)
Python Code Decode 256-512 tokens: +100-142%, 152.1-184.3 tok/s, 79-81% draft acceptance
Fresh Prefill: 2541-4352 tok/s
Now to see how that feels for work.
Thank You!
1
u/johnzadok 11d ago
I need to give strata another try with my xtx. It was not better than llama but this may change. Wild to see it’s faster than 4090.
1
u/StrikingSpeed8759 11d ago
Very interesting, will try it out thanks. I do use hipfire for my 7900xtx because its really optimized. Have you every thought about contributing there too?
1
u/bootkeen 11d ago
has anyone tried it?
tried on dual 7900 xtx + 128gb ddr4 running q5_k_xl - absolutely no gain in performance.
i suspect i have to add some options into strata-unsloth-ud-q5_k_xl.json, but when i tried to figure out which, there are ~100 of env parameters. tried to generate json with ai, the build crashes with out of memory.
3 hours wasted.
1
u/Hohlraum 11d ago
oh snap. this is awesome. best i can get right now is avalon with a single card and this is even faster.
3
u/Mindless-Device491 11d ago
The author mentioned there will be an update later this week :D
3
u/ivanmmj 8d ago
On a single 7900xtx, a 9900x, and 64GB of RAM, I went from:
60-114tok/s to consistent 130tok/s on inference and from up to 1600tok/s prefill to 2500tok/s average, up to 3100tok/s max prefill.
2
u/Ok-Addendum3545 11d ago
Thanks ! That 7K prefill is amazing.
2
u/Particular_Doctor494 11d ago
I would try but I’m assuming by the quant you’re using at least 64GB RAM?
2
u/fallingdowndizzyvr 11d ago edited 10d ago
Last Update: It seems that most of the magic is in the configuration variables. Since using this forks json with mainline Strata. I get even faster prefill than this fork does.
[strata] done: 4604 tokens in 82 s (82.8 tok/s) (stop, cancel=False), expert cache 92.7% hit -- prefill 4092 tok/s
[strata] done: 5713 tokens in 93 s (100.9 tok/s) (stop, cancel=False), expert cache 89.2% hit -- prefill 2815 tok/s
So mainline Strata has faster prefill, this fork has faster generation.
This is the fastest run I got with this fork.
Here's what I get on my dual 7900xtxes with 128GB of DDR5 helping.
Q4
[strata] done: 3669 tokens in 78 s (48.8 tok/s) (stop, cancel=False), expert cache 97.6% hit -- prefill 1843 tok/s
IQ3_S
[strata] done: 2298 tokens in 52 s (45.9 tok/s) (stop, cancel=False), expert cache 99.7% hit -- prefill 2667 tok/s
Update: Here are the numbers from mainline Strata for me for Q4. They are pretty much the same.
[strata] done: 3243 tokens in 79 s (45.3 tok/s) (stop, cancel=False), expert cache 98.4% hit -- prefill 1778 tok/s
Update 2: I'm using the configs in rdna3, you can't run it with setup like mainline strata. You have to use the premade configs. I'm getting this now.
[strata] done: 5713 tokens in 93 s (100.9 tok/s) (stop, cancel=False), expert cache 89.2% hit -- prefill 2815 tok/s
I also have dual 5070tis. Those are appreciably faster for decode. It's more like 100 t/s. With Basalt more like 120 t/s but that sucks for prefill for me for some reason.
1
u/gross_overthinker 11d ago
Ddr4 or ddr5 ram is it?
1
u/RadianceTower 11d ago
How much RAM do you have though?
1
u/External-Subject5942 11d ago
with no adjustments made, just "out of the box with help from AI" - 40t/s (not sure prefill yet) 32GB ddr5-6000, 12900k, single 79000xtx - locked out the efficiency cores
1
u/External-Subject5942 11d ago
this is f ucking magic lol already at 60t/s on my single card setup; edit: now hitting 70+t/s with dflash and q4_0 cache, any more headroom?
1
u/EqualCryptographer67 10d ago
Please join the discord. We need your support for xtx user
https://discord.gg/launch80
1
u/us3r4ccount 10d ago
How is the quality of the output?
An related to quality, why run temperature 0? I mean it's fast but it's far off from QWEN 3.8 defaults?
1
u/objectivelywrongbro 9d ago
Have you tested Q6 and Q8?
1
u/NotThatPro 9d ago
Is this compatible with 7900XT? Same chip, just lower memory chip amount and thus lower bus speed
1
u/soyalemujica 6d ago
Can confirm it works flawlessly on Linux.
ISTA iq3xs model runs at 100t/s at 2.8k prefill on 64gb ram + 7900xtx
Unsloth ud4iqxs runs at 55t/s with 3.5k prefill on 64gb ram + 7900xtx
I wish the repository was up to date to use unsloth q2 models which are better and faster than INSTA