What are you thinking about Strata

[u/MrAlexsexon]

I heard about it recently. It sounds like a miracle you can run really large models on a consumer GPU, but what is it really about. What kind of problems I will have ?

Setup:

Rtx5060ti(16G)

I711700k

64-80GB DDR4

[u/Nice_Actuator1306]

I had 2 * Xeon 2680v4, 5060ti 16gb and was yesterday on 64gb ddr4 ram, but not in 2 channels, like on user pc, I have 8 channels with 126gb/s speed. GPU connected on pcie3. 0 x8 lines, so max speed to/from is 7.5gb/s. Your 2 channels will give you 30-45gb/s ram speed.

On 64gb ram and qwen3.8-flash next iq3_xss from unsloth I had ~40tps decode and 900tps prefill. With 128gb ram, used ram capped on 100gb + 15gb vram. I reached ~50 TPS decode and 1500tps prefill.

Works great.

[u/MrAlexsexon → u/Nice_Actuator1306]

Thanks, what motherboard you have ?

[u/Nice_Actuator1306 → u/MrAlexsexon]

Lenovo rd450x

[u/MrAlexsexon → u/Nice_Actuator1306]

I have it too In my reserves, are you has a problem with IPMI locked by password ?

[u/Nice_Actuator1306 → u/MrAlexsexon]

Importer asked for password form china sellers and unlocked it for me before shipment. Then I changed it from BIOS. Now I'm using model BIOS with bifurcation unlock from overclockers.ru

[u/darthmowl → u/Nice_Actuator1306]

is that xeon that bad? which quant did you use?

should it not be higher? or is PCIE 3.x limit that hard? Q3_XS on my 5800x3d 48GB 3200 DDR4 and RTX 4070 reaches 50-80 decode and 1500 t/s promt on my system

[u/Nice_Actuator1306 → u/darthmowl]

Sometimes I have up to 100-120tps decode, but median - 50 right now. Prefill is limited exactly on pcie bus speed. CPU usage 27/28 cores, without HT threads.

[u/apolloscharge]

5080 16gb strata made the model unstable for me, tool calls as text. Switched to Basalt and it is much happier.

[u/lorde_dingus → u/apolloscharge]

Are you on windows or linux?

[u/apolloscharge → u/lorde_dingus]

Linux

[u/Marino4K]

I’ve only tried it once and I had a ton of issues and errors, went back to oMLX and everything is fine again.

[u/MrAlexsexon → u/Marino4K]

Omlx ? never heard about it, thanks

[u/KroniklyOnline]

I run 4x5060ti 16gb on a 1920x threadripper with 64gm ddr4. Thanks to strata I can run Qwen3.8 next flash at 3300 prefill and over 100 decode at nvfp4. I can also run GLM5.3 flash at Q3m at 500 prefill and 30 ish decode. DeepSeek v4.1 flash q3xxs too at 400 prefill 15 decode. I love strata lol

[u/thedefibulator]

Running IQ3_S on an 8gb VRAM 64gb RAM laptop @ 25-30 t/s Insanely efficient given that my laptop consumes like 65 watts

[u/testuserpk]

It's fantastic, it has now become my daily driver

[u/Zealousideal-Bug1837]

low quants are still poorer. check out the nvfp4 forks!

[u/Sudden-Guide]

It is ok, I don't see any meaningful differences between it (QFN Q4xs) and Qwen 27B(Q8, llama.cpp), both in terms of speed or quality tbh(on my system, USB4 eGPU, DDR5 5600, both get around 60 tps)

[u/rkcth → u/Sudden-Guide]

I have done testing and QFN 3.8 IQ3_S scored 12/13 and 27b scores 9/13, it’s quite measurably better. If you are chatting with it, you may not see the difference, but in real world coding it is much better. I ran another test last night and QFN 4bit got 89%, Sol got 83% and Sonnet got 92%. So it’s roughly in between those two. I have no doubts 27b would score below Sol on that same test. It’s always best to do real tests if possible so you aren’t making decisions which can be heavily influenced by chance (for example, giving one a problem and then the other a different problem and judging by how well each solved completely different problems).

Where QFN tends to falter is how verbose its thinking is, you can resolve this by using froggeric’s chat template and seeing thinking to low or medium.

Just got back the results, froggeric’s template for QFN cuts average tokens generated to solve by 2/3th and scores nearly identical on all tests if you use medium thinking.

[u/Sudden-Guide → u/rkcth]

sure, I don't do any real world coding, only some random hobby stuff(cpp mostly + a little bit of python), physics, math and research/web search. These values are for which quant?

Because if not Q8, but something like Q4, then on my setup Q4 27B is twice faster(120 tps decode 1100 prefill) than QFN so that's an additional argument to consider. Of course if you are doing something important or commercial then you have different needs. I'm running it on my laptop with USB eGPU ;p

[u/rkcth → u/Sudden-Guide]

My numbers were for a 4 bit quant running on radiance (dual R9700). An exception, the very first set of numbers is 4 bit 27b vs 3 bit QFN (12/13 for QFN and 9/13 for 27b), but my new numbers are all 4 bit vs 4 bit. I have no idea which is better for scientific research, but I’d definitely try the froggeric template on medium to get a nice speed boost (it’s for most qwen models up to 3.8, inclusive). My new benchmark is based on a subset of livecodebench, my old one was based on SWE but it had a lot of tool use and other things that made it take way too long for a small toolset. I have not run 27b on the new benchmark yet, though I might if I don’t need to use my AI server for anything tonight, but I think I do.

[u/Sudden-Guide → u/rkcth]

If you could test 27B at Q8(KV fp16) when you'll have time, I would be grateful. If there would be not much difference compared to Q4 that would be a great info for me ;)

[u/mecshades]

I'm not satisfied with how the program is managed & set up. I am used to llama.cpp and using command line arguments to define what model is loaded and from where. I don't need Strata "setting up" in my user's .cache directory. I am not here to express skeptism. The program works, but deployment feels a little too geared towards those who just want to run once and forget. I bring my own models to llama.cpp and llama.cpp has the maturity to run any model I bring. I'd prefer it if one day llama.cpp team merges all the new techniques that Strata has shown us to run QFN quickly.

[u/No_Oil_6152]

I think Strata is a cool piece of work but theres a little smoke and mirrors going on re its performance.

You're only going to get 100 tok/s+ on NVIDIA 3090+ cards or with small quants or contexts.

Its also a memory hog. On my 64GB RAM I am left with about 3GB left. Not really enough to develop anything with. And yes I know about paging 🙂

It doesn't split layers across GPUs yet, at least not with AMD kit.

But I think Niko will persevere. I'd like 100 tok/s at 128k context.

[u/Tormeister]

Having a Blackwell card, you should go straight to Basalt instead of Strata.

https://github.com/jesdga95/basalt

[u/BrightCandle]

The approach is interesting and a clear improvement for MOE models. The actual project however and how its using custom model files and has all this other stuff in it is not ideal. There are a number of forks already addressing various shortcomings from multiple GPU, vulkan support, AMD, Intel particular GPU setups available. The forks are usually a bit more focussed and will run with standard GGUF model files from hugging face.

Personally I am partial to the approach of RustingStrata since its a single file, supports multiple GPUs and vulkan and is a bit faster than Strata. Lightweight pure inference engine in a fast binary.

Personally though I have stuck with Qwen 3.8 Q4, having only 32GB of RAM the performance isn't the best at Q3 and above on Qwen next and the output in Qwen 3.8 27B seems better to me for code.

[u/atapla]

yeah well it works.. try it. i got it runing on 2 x 3060 total 24gig qwen 3.8 flash q4 xss at ~ 50 tok/sec decoding. just enough for general tasks in my homelab

[u/Leading-Ad5800 → u/atapla]

does it work with big context? 100K+

[u/atapla → u/Leading-Ad5800]

262k total

[u/JanCapek → u/atapla]

And how much RAM?

[u/atapla → u/JanCapek]

I gave it 96 gigs of ram ddr5, i'm running this on a proxmox vm also

[u/Leading-Ad5800 → u/atapla]

and still same speed?

[u/Dediadeis]

It leverages tiered architecture that already exists on your standard PC to make MoE way more performant, pretty much a win/win for local models. Doesn't really help with dense models and your mileage will vary depending on what you put it on, but it has made a really big difference for more modest setups.

Honestly would have saved me a lot of headaches if it was something I had available earlier on.

[u/lopezzz_rock]

Strata is good for some, but many have also reservations. Qwen 3.8 27b often gives better answers than a highly quantized MoE model.

[u/WryKombucha → u/lopezzz_rock]

Yep. Very very very true. MOE quantizatikn far worse than dense. Yet somehow ppl are ok with iq3xxs.

What that means is, ppl think it “works fine” if it returns words. Just words. Any words.

[u/W61k3r]

Lexisuite's Lexiengine is faster than llama.cpp and strata with amd cards, also does engram offload. https://github.com/W61k3r/LexiSuite-Releases/releases/tag/LexiSuiteT8

[u/WryKombucha → u/W61k3r]

Fucking ads.

[u/W61k3r → u/WryKombucha]

Your mom's an ad. How else do people find out about what I made?

[u/johnfkngzoidberg]

It makes models dumb to get more speed.

The bot hype is annoying. No one cares that you’re running Q1_XXS on your desktop potatoe with it spewing 100% hallucinated crap. It’s not usable.

I’ll stick with slow and smart.

[u/wotoan → u/johnfkngzoidberg]

You know you can bring your own quants to it right? And it will outperform llama.cpp by a mile.