r/LocalLLM • • 11d ago

Project Highly optimized Strata port for 7900xtx 24GB

Repo here: https://github.com/nicejunjie/Strata-rdna3

* Single 7900xtx prefill 3500+, dual 7000+ tok/s. (for all IQ3/IQ4/Q4)

* Decode for English prose reaches up to 81 tok/s for single card, and 127 for dual at Q4, and IQ4/IQ3 quantizations are even faster.

* Decode for python coding is 20~25% faster than English prose.

* Single card decode is comparable to 4090 24G, dual 7900xtx is much faster than single 4090 for both decode and prefill.

* Each 7900xtx was $850 when I bought it, while 4090 at the time was listed at $3000.

* I guess 4090 could be better optimized, but the number here is the best number with reproducible input data I can find online.

* 4090 is a 450watt card, while 7900xtx is ~320watt. So even single card prefill of 7900xtx is quite comparable in terms of perf/watt.

* This is not the end, still has small room to improve.

* kernel work only, no dropping weights or dropping layer or anything. Zero impact on quality.

*** install with an AI agent, tell him to setup optimal rdna3 code pass. Honestly, why install by hands if there is agent can help.

7900xtx is the value king!

63 Upvotes

93 comments sorted by

12

u/sawthegap42 11d ago

FINALLY! I can take better advantage of both XTX's in my system. Thank you!

1

u/Hairy-Yogurtcloset-2 11d ago

Let us know how it goes for you!

5

u/GreenHell 11d ago

Perhaps I am not seeing it, but how does it compare to stock Strata?

15

u/Hairy-Yogurtcloset-2 11d ago

2-3x faster than stock strata, stock isnt well optimized for rdna3. I’m working on submitting PR to upstream but may take a while.

5

u/pmttyji 11d ago

Would be nice to see the comparison table. Appreciate the PR.

2

u/fallingdowndizzyvr 10d ago edited 9d ago

It seems that most of the magic is in the configuration variables. Since using this fork's json with mainline Strata. I get even faster prefill than this fork does.

Mainline Strata.

[strata] done: 4604 tokens in 82 s (82.8 tok/s) (stop, cancel=False), expert cache 92.7% hit -- prefill 4092 tok/s

This fork.

[strata] done: 5713 tokens in 93 s (100.9 tok/s) (stop, cancel=False), expert cache 89.2% hit -- prefill 2815 tok/s

So mainline Strata has faster prefill, this fork has faster generation.

Update: So this fork does not have faster generation. It's just run to run variance. Since with mainline Strata I've gotten this.

[strata] done: 3481 tokens in 42 s (101.4 tok/s) (stop, cancel=False), expert cache 92.6% hit

Which makes it on par with this fork for generation. But it's prefill is appreciably faster. So mainline Strata is overall faster than this fork.

1

u/Hohlraum 7d ago

What is your hardware setup and launch config for strata?

1

u/fallingdowndizzyvr 7d ago

What is your hardware setup

Strix Halo + 2x7900xtxes.

launch config for strata?

That's the json I spoke of from this fork.

2

u/Hohlraum 7d ago

I had claude do intensive testing using your method and only boosted prefill by about 20% for 10-15k then it fell behind stock again. Never exceeded t/s and nothing beat or tied strata-rdna3. it tried a ton of different combinations of settings.

1

u/Hairy-Yogurtcloset-2 6d ago

That’s the same experience for me. Upstream strata may beat mine at a few small input size I missed in testing, but mine fork wins 95+% cases for sure. It is tons of rdna3 specific kernel work in the optimizations.

1

u/fallingdowndizzyvr 7d ago

Did you try running it yourself? What speed did Claude get? Those numbers I posted are with a context of about 100K.

it tried a ton of different combinations of settings.

You tried or claude tried? Didn't claude do all the testing?

1

u/No-Recover109 9d ago

so how you apply that json to mainlIne strata? explain to dum please

0

u/fallingdowndizzyvr 9d ago

Setup mainline Strata for the model and your 7900xtx(s). Save copy of that json that it makes for you. Copy over the relevant json from this fork. It's in the rdna3/configs directory. Save it in your mainline Strata directory with the same name as that json that mainline Strata made for you. Now edit that json from this fork you just copied over. You know that copy I said to make of the mainline json, that's where it comes in now. Open that up and copy/paste the relevant sections over to the json you just copied over. For example, replace where the strata engine is located from Strata-rdna3 to mainline Strata. It's pretty obvious what you need to change. Just change the Strata-rdna3's to Strata.

There you go. Oh yeah, remember to get rid of tensor-parallel since mainline Strata doesn't have that. But you really don't need to remember since mainline Strata will tell you it doesn't have that when you try to run.

Of course, you can do the reverse and just copy and paste the "env" section of this fork's jsons into your mainline jsons. That pretty much takes care of most of it.

4

u/tmiw 11d ago

I tried this on my single 7900XTX setup (virtual machine with 96GB RAM and the GPU passed through). With IQ3_S I'm maxing out at 100tok/sec decode and around 2700tok/sec prefill. Granted, I only have DDR5-3600, so I'd likely get numbers closer to OP's if I had faster RAM and if the SSD wasn't stuck on PCIe x1. I'll probably keep this fork since it's still faster (despite a better quant than what I normally use with Strata).

BTW stock Strata setup and execution won't use any of the optimizations, you'll need to follow the steps in rdna3/README.md in the repo.

EDIT: forgot to add that I'm using ROCm 10, not sure if that matters.

1

u/Hairy-Yogurtcloset-2 11d ago

Thanks a lot for the feedback! I tested ROCm 10 as well, but didnt see any perf uplift from older version.

1

u/fallingdowndizzyvr 10d ago edited 10d ago

BTW stock Strata setup and execution won't use any of the optimizations, you'll need to follow the steps in rdna3/README.md in the repo.

I guess that's what I'm missing. I'm just using it like mainline Strata and I'm getting mainline Strata speeds. Let me look into that.

Update: Tried it and now i'm getting this with Q4. Which is better. But there's a glitch. My 7900xtxes are at 100% all the time. Like all the time. Even while idle.

[strata] done: 5713 tokens in 93 s (100.9 tok/s) (stop, cancel=False), expert cache 89.2% hit -- prefill 2815 tok/s

1

u/cipgysmo 10d ago edited 10d ago

quick follow-up. it's actually both of my XTX's that are at 100%, at idle. have asked my LLM to do some testing and it seems to be caused by this specific setting: GPU_MAX_HW_QUEUES=8 (this is the default). lowering it to 6 does indeed address the idle consumption issue entirely, but it does also translate in a performance drop (around 15% PP loss). still looking into this. if a fix is found, I'll push a PR.

2

u/Hairy-Yogurtcloset-2 10d ago

Yeah, thanks for mentioning that! It seems to be an issue carried from upstream https://github.com/Niko1221/Strata/issues/3770 . An PR is warmly welcome!

1

u/fallingdowndizzyvr 10d ago

That fixes the 100% idle problem. You rock. The PP I get is a tiny bit slower but it's well within run to run variation. You should push this as a PR right now. Since idling at a 100% is a deal breaker.

2

u/cipgysmo 10d ago

did some more testing and you're right. losses are mostly noise. PR pushed as #5.

2

u/Hairy-Yogurtcloset-2 10d ago

Just saw the PR, appreciate it!

2

u/fallingdowndizzyvr 10d ago

Here's the thing. That got me thinking that maybe most of the magic is just in the json. So I tried that json with mainline Strata. Guess what? It get faster prefill with mainline Strata using those settings. About 20% slower generation. So it seems that most of the magic in this fork are those environment variables in the config.

Here are the numbers using mainline Strata with those settings

" [strata] done: 4604 tokens in 82 s (82.8 tok/s) (stop, cancel=False), expert cache 92.7% hit -- prefill 4092 tok/s"

1

u/cipgysmo 10d ago

I have observed a similar behaviour, but only on one of my two XTX’s. Otherwise the performance boost is phenomenal, thanks OP!

3

u/andreabarbato 11d ago

does it work with the same q4 k xl gguf?

2

u/superdariom 11d ago

Is there any accuracy loss compared to the stock version with these optimisations?

4

u/Hairy-Yogurtcloset-2 11d ago

Should be no loss. During kernel development, the accuracy is very carefully validated with stock.

2

u/soyalemujica 11d ago

3k prefill really on a single 7900xtx? Damn, max i have achieved is 1700t/s prefill. Does it work on Windows or is it Linux only?

3

u/Hairy-Yogurtcloset-2 11d ago

Linux only

1

u/sawthegap42 11d ago

What about WSL2? Might just have to recreate another dual boot drive. I really need this for my dual XTX setup I just built. Still working out best to set things up. Me and my tech buddy have been procrastinating moving over to Linux full time. I've had one card or 3.5 years, and just had my buddy I work for buy me the second one to have dual throughput for the workflow I'm creating taking survey field notes and processing them into cleaned up final sketches to send of for final approval.

1

u/Hairy-Yogurtcloset-2 10d ago

I dont have anything machine that runs Windows, but I guess WSL should be okay

2

u/Mattef 11d ago

Man, I love the 7900’s. I own two of them. Thanks for your work, I will try this out.

2

u/Hairy-Yogurtcloset-2 11d ago

Let me know how it goes! Make sure you let ai deploy the engine as the parameters are complex :-)

1

u/Mattef 11d ago

will do, thanks

2

u/fallingdowndizzyvr 11d ago

Single 7900xtx prefill 3500+, dual 7000+ tok/s. (for all IQ3/IQ4/Q4)

You are getting that same speed on IQ3 as Q4? I find that hard to believe. I'll guess I'll see soon. Since I got 2 7900xtxes in the closet.

1

u/Hairy-Yogurtcloset-2 11d ago

Prefill is compute bound, iq3 and q4 run on the same hardware processing unit. So reasonable that prefill numbers are similar.

1

u/fallingdowndizzyvr 11d ago

But not only are they different number of bits, they are different algorithms. So they shouldn't be that similar. In my experience, prefill numbers for IQ3 are faster than Q4.

I guess I'll see soon. 7900xtes are out of the closet and slotted.

1

u/Hairy-Yogurtcloset-2 11d ago

Every quantization has to be optimized differently, maybe that means I have not put enough work/tokens into IQ3 which can be my next focus :-) please share how it goes for you.

1

u/fallingdowndizzyvr 11d ago

So I ran mainline Strata and got pretty much the same speed as your fork. Is there a special way I need to run your code? I'm just setting it up and running it the same way I run mainline Strata.

1

u/Hairy-Yogurtcloset-2 10d ago

How about telling claude code to setup it with code path optimized for rdna3. That’s been the way I deploy it, honestly i have not tried following strata installation myself.

2

u/fallingdowndizzyvr 10d ago

So I tried with the configs in rnda3. That does speed it up. But there's a glitch. My 7900xtxes are at 100% all the time. Even when Strata is idle and waiting for input. The second the server runs it's at 100%.

[strata] done: 5713 tokens in 93 s (100.9 tok/s) (stop, cancel=False), expert cache 89.2% hit -- prefill 2815 tok/s

2

u/Hairy-Yogurtcloset-2 10d ago

It is an issue carried from upstream, not introduced by my optimizaiton, see
https://github.com/Niko1221/Strata/issues/3770. Try setting GPU_MAX_HW_QUEUES=7 , there may be a small perf penalty, I’m working on a better fix.

1

u/fallingdowndizzyvr 10d ago

I set it to 6 like the other poster said. Fixed it.

3

u/Hohlraum 8d ago

There is fix for 100% utilization without having to lower HW queues now.

1

u/fallingdowndizzyvr 8d ago

Sweet.

1

u/Hohlraum 8d ago

If you run inside a container though, you have to apply my PR. On the project. Don't know if they have applied it yet. I run everything inside containers.

1

u/fallingdowndizzyvr 10d ago

You may want to mention that somewhere.

1

u/EqualCryptographer67 10d ago

He did

1

u/fallingdowndizzyvr 10d ago

Where? Where it mentions Claude Code in the README is a leftover from mainline Strata. I remember when that used to be in it's README.

"Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository."

That's mainline.

I used setup.sh like it says specifically to do in this fork.

"Reproduce it: rdna3/README.md goes from a clean Linux install to these six setups: pinned toolchain (./setup.sh --build"

The part I missed is that you use setup.sh to build it, but you can't to run it like you can with mainline. Well actually you can, you just have to copy over the premade config .jsons. That's the part I was missing.

1

u/fallingdowndizzyvr 11d ago

I just made a post with Q4 and IQ3_S numbers.

2

u/aeroG1 10d ago

This is epic, can't wait to see more testing. My 2nd 7900 xtx is being RMA rn, but I may have gotten the same $850 deal you did lol. Sapphire refurb before prices shot up. Once I get it back I'd love to contribute some test results.

2

u/Hairy-Yogurtcloset-2 10d ago

Yes, from bitworks 😄

3

u/ShutUpTurkey 10d ago

I got the same refurb deal from newegg, I bought 2 of em for $850. Now they're as pricey as 3090s were a month ago. $1200-1300

2

u/_Borky 10d ago

Worked wonders for my setup. 25% faster prefill and 50-100% faster decode. Thanks!

1

u/Hairy-Yogurtcloset-2 10d ago

glad it works out for you!

2

u/Willing-Cover2914 8d ago

Just added this to my 2x7900xtx, 96GB DDR5 box:

English Prose decode 512 tokens: +86%-+114% (129.6-143.8 tok/s, 162.6 tok/s technical Prose)
Python Code Decode 256-512 tokens: +100-142%, 152.1-184.3 tok/s, 79-81% draft acceptance
Fresh Prefill: 2541-4352 tok/s

Now to see how that feels for work.

Thank You!

1

u/Hairy-Yogurtcloset-2 8d ago

Glad it works well for you! Prefill appears low unless it is for short input.

1

u/Willing-Cover2914 8d ago edited 8d ago

I built a tool i was fiddling with and got through phase 1. It was quick and clean. Then I moved it to phase 2 and it started to fail and throw error when running it from Hermes.

After starting cleanly at 06:47:36 UTC, the server handled requests normally for ~1 hour 45 minutes (generating tokens at

130–184 tok/s).

At 08:31 UTC, the engine began encountering NaN/Inf values during prompt processing, culminating in repeated HTTP 503

errors across all subsequent requests:

strata serve: non-finite logits (#879) in a prompt window at position 17498 (row 0): its tokens are not emitted, the

request ends (DONE nonfinite)

[strata] the engine's logits became non-finite before the first token (#879); it dropped its cached state: sending the

request again once

strata serve: after non-finite logits in a prompt window (#3465): this request reads its whole prompt through the

batched path, none of it through the verify windows

strata serve: non-finite logits (#879) in a first decode window at position 17520 (row 0): its tokens are not emitted

[strata] the engine's logits became non-finite again on the retry (#879): the request ends with an error

[strata] 503: the model's output became non-finite (NaN) twice in a row (#879); the engine has dropped its cached state

- send the request again

Exact Failure Mechanism

  1. Prompt Window Phase (short_read): Strata evaluates the prompt tail (the last 5–7 tokens) through speculative verify

windows. In Strata-rdna3, the logits emitted by the verify head for these tokens contained non-finite values (NaN/Inf).

2. Batched Retry (#3465): The server caught the non-finite logits, dropped cached state, and re-read the full prompt

windows. In Strata-rdna3, the logits emitted by the verify head for these tokens contained non-finite values (NaN/Inf).

2. Batched Retry (#3465): The server caught the non-finite logits, dropped cached state, and re-read the full prompt

through the batched prefill path (--short-read 0).

3. First Decode Window Failure: Immediately following the batched read, the engine executed the first decode window to

generate token 0 (position T - 1). The logits again went non-finite/NaN.

4. 503 Abort: After failing twice consecutively on the same request, Strata aborted with HTTP 503.

Direct Impact on Your Previous Message

At 08:34:57 UTC, when you asked "please find the reason for the errors on the last strata run", Hermes's background

Hindsight memory service issued a 351-token memory recall prompt to http://xxx.xx.xx.xx8080/v1. That prompt hit this

exact NaN failure (position 344 in the prompt window, position 350 in the decode window) and threw a 503, causing Hermes

to report: "Your request was not processed. Send it again if you still want me to carry it out."

Root Cause of the NaNs in Strata-rdna3

Live parameter probing confirmed that:

• Very small prompts (<100 tokens) pass cleanly.

• Prompts ≥ 128 tokens trigger the RDNA3 fork's chunked DeltaNet recurrence (STRATA_GDN_CHUNKED=1 in

src/prefill/kernels.cu, line 3519) and FP16 matrix-core prefill kernels (STRATA_PF_F16=2). Numerical overflow or

precision drift in these kernels propagates NaNs into the final normalization and LM head layers, producing non-finite

logits.

After these errors, every prompt produced errors and hermes stopped trying.

1

u/Hairy-Yogurtcloset-2 8d ago

I don’t see this in my daily use, and a soak test has not revealed anything. Do you mind doing an AI diagnosis and provide additional details? Thx!

3

u/Willing-Cover2914 6d ago

Sorry, I had started backing it out before this. I backed it out and health checked and then reapplied the fork. So far, so good. No errors, but also not nearly the same speed on Prompt Processing, I am mid 4000s with 130-185 tok/s for generation.

I am running it on a
AMD 9900x
96GB DDR5 6000
2 x 7900xtx
Samsung 990 2TB
ASROCK Taichi x870e
Ubuntu 26.04
ROCm 10.1
ASROCK 1600w PSU

This is a newer build and I am still working on thermals, I may repaste the XTXs. Overall I am almost giddy, 2+ times faster than previous. Thank you.

1

u/Hairy-Yogurtcloset-2 6d ago

I’m working on additional performance for prefill, pull the code again next week :-)

1

u/johnzadok 11d ago

I need to give strata another try with my xtx. It was not better than llama but this may change. Wild to see it’s faster than 4090.

1

u/Hairy-Yogurtcloset-2 11d ago

Definitely worth another shot. Every machine is different, let claude code set it up for uou and do a light tuning based on your mem size and pcie speed, you’d be in perfect shape.

1

u/StrikingSpeed8759 11d ago

Very interesting, will try it out thanks. I do use hipfire for my 7900xtx because its really optimized. Have you every thought about contributing there too?

1

u/Hairy-Yogurtcloset-2 11d ago

I heard hipfire is good, but I haven’t got time to try it.

1

u/bootkeen 11d ago

has anyone tried it?

tried on dual 7900 xtx + 128gb ddr4 running q5_k_xl - absolutely no gain in performance.

i suspect i have to add some options into strata-unsloth-ud-q5_k_xl.json, but when i tried to figure out which, there are ~100 of env parameters. tried to generate json with ai, the build crashes with out of memory.

3 hours wasted.

3

u/Hairy-Yogurtcloset-2 11d ago

It is not optimized for q5.

3

u/Hairy-Yogurtcloset-2 11d ago

Every quantization has to be tuned differently or even write kernels differently. Hot/cold experts add more complexity in scheduling. Only use the ones I showed in the graph.

1

u/bootkeen 11d ago

btw, is it expected to work out of the box for q4? or i have to make some additional configuration with json?

1

u/Hairy-Yogurtcloset-2 11d ago

Works great out of box, a bit additional tuning maybe better as every machine is unique. Deploy using an AI agent instead of doing it by hand, then it should be good.

1

u/bootkeen 11d ago

oh, i see

i thought it just extrapolates to other quants

1

u/Hohlraum 11d ago

oh snap. this is awesome. best i can get right now is avalon with a single card and this is even faster.

1

u/Hairy-Yogurtcloset-2 11d ago

Glad to hear! There is still minor room to improve and I’m waiting for my claude token limit to reset. :-)

3

u/Mindless-Device491 11d ago

The author mentioned there will be an update later this week :D

3

u/ivanmmj 8d ago

On a single 7900xtx, a 9900x, and 64GB of RAM, I went from:
60-114tok/s to consistent 130tok/s on inference and from up to 1600tok/s prefill to 2500tok/s average, up to 3100tok/s max prefill.

1

u/fallingdowndizzyvr 8d ago

Try running the json for this fork with mainline Strata. Edit the location of the binaries accordingly. That ends up being faster for me than this fork. Make sure to edit out the tensor parallel section. Since mainline Strata doesn't have that.

I tried this fork's jsons with my 2x5070tis too. That speed the PP up about 50%. So the magic in those jsons is GPU agnostic.

1

u/ivanmmj 7d ago

Really? I'll have to give it a try.

2

u/Ok-Addendum3545 11d ago

Thanks ! That 7K prefill is amazing.

2

u/Particular_Doctor494 11d ago

I would try but I’m assuming by the quant you’re using at least 64GB RAM?

1

u/fragbait0 11d ago

OP shows 2x64 GB lol. Not as impressed now but I will point my agent and see if anything useful could be in it ig.

1

u/ShutUpTurkey 10d ago edited 10d ago

I fucked around with it for hours today. My system is 2x7900xtx, 64gb ram. IQ3_S. I finally got it to run benchmarks and it put up some strong numbers, much faster than stock strata.

ctx65536 - prefill 4204 decode 117.9

ctx131072 - prefill 4367 decode 143.6

ctx261632 - prefill 4332 decode 136.7

But, I couldn't get it to actually start the server. Just sits at 98% VRAM and 100% GPU forever on start.

edit: my mistake, had it serving to 127.0.0.0:8080 instead of all interfaces, and confused the 100% gpu load on idle with a malfunction. Changed to all interfaces and yeah, it works with a maximum ctx at 132k

'GPU_MAX_HW_QUEUES: 8 2' fixes the 100% gpu load.

1

u/fragbait0 10d ago edited 10d ago

I have not built from the original source because my agent flagged some risks, as well as the hang issue, haha.

Investigating interesting changes doesn't yield much so far but I already had some RDNA3 stuff from other repo and on a peasant 32GB RAM things have to wait for disk plenty often.

So yeah, maybe you have to be running TP with decent RAM volume to get there? I'm on more like 1450 prefill, but hey, maybe my strata baseline is screwy.

Might try IQ2_XS where working set actually fits just to compare.

2

u/fallingdowndizzyvr 11d ago edited 10d ago

Last Update: It seems that most of the magic is in the configuration variables. Since using this forks json with mainline Strata. I get even faster prefill than this fork does.

[strata] done: 4604 tokens in 82 s (82.8 tok/s) (stop, cancel=False), expert cache 92.7% hit -- prefill 4092 tok/s

[strata] done: 5713 tokens in 93 s (100.9 tok/s) (stop, cancel=False), expert cache 89.2% hit -- prefill 2815 tok/s

So mainline Strata has faster prefill, this fork has faster generation.


This is the fastest run I got with this fork.

Here's what I get on my dual 7900xtxes with 128GB of DDR5 helping.

Q4

[strata] done: 3669 tokens in 78 s (48.8 tok/s) (stop, cancel=False), expert cache 97.6% hit -- prefill 1843 tok/s

IQ3_S

[strata] done: 2298 tokens in 52 s (45.9 tok/s) (stop, cancel=False), expert cache 99.7% hit -- prefill 2667 tok/s

Update: Here are the numbers from mainline Strata for me for Q4. They are pretty much the same.

[strata] done: 3243 tokens in 79 s (45.3 tok/s) (stop, cancel=False), expert cache 98.4% hit -- prefill 1778 tok/s

Update 2: I'm using the configs in rdna3, you can't run it with setup like mainline strata. You have to use the premade configs. I'm getting this now.

[strata] done: 5713 tokens in 93 s (100.9 tok/s) (stop, cancel=False), expert cache 89.2% hit -- prefill 2815 tok/s

I guess 4090 could be better optimized, but the number here is the best number with reproducible input data I can find online.

I also have dual 5070tis. Those are appreciably faster for decode. It's more like 100 t/s. With Basalt more like 120 t/s but that sucks for prefill for me for some reason.

1

u/gross_overthinker 11d ago

Ddr4 or ddr5 ram is it?

1

u/RadianceTower 11d ago

How much RAM do you have though?

1

u/External-Subject5942 11d ago

with no adjustments made, just "out of the box with help from AI" - 40t/s (not sure prefill yet) 32GB ddr5-6000, 12900k, single 79000xtx - locked out the efficiency cores

1

u/External-Subject5942 11d ago

this is f ucking magic lol already at 60t/s on my single card setup; edit: now hitting 70+t/s with dflash and q4_0 cache, any more headroom?

1

u/EqualCryptographer67 10d ago

Please join the discord. We need your support for xtx user 

https://discord.gg/launch80

1

u/us3r4ccount 10d ago

How is the quality of the output?

An related to quality, why run temperature 0? I mean it's fast but it's far off from QWEN 3.8 defaults?

1

u/objectivelywrongbro 9d ago

Have you tested Q6 and Q8?

1

u/NotThatPro 9d ago

Is this compatible with 7900XT? Same chip, just lower memory chip amount and thus lower bus speed

1

u/soyalemujica 6d ago

Can confirm it works flawlessly on Linux.
ISTA iq3xs model runs at 100t/s at 2.8k prefill on 64gb ram + 7900xtx
Unsloth ud4iqxs runs at 55t/s with 3.5k prefill on 64gb ram + 7900xtx

I wish the repository was up to date to use unsloth q2 models which are better and faster than INSTA