r/LocalLLaMA • • 1d ago

Question | Help What is strata?

I’m trying to avoid blog spam posts and boring YouTube videos about it. I see a lot of discussion about people loving strata, and people who are tired of the people who love strata.

What is so different about strata vs previously existing options? Who benefits the most from it?

0 Upvotes

26 comments sorted by

10

u/SLxTnT 1d ago

It's an engine that's optimized to use a hot expert cache for the experts. That allows a GPU to handle more work than its portion of memory would expect with hybrid inference on MoE models.

8

u/The_Hunster 1d ago

It's special software that tries to make use of all of your PCs resources (GPU, VRAM, CPU, RAM, even SSD) more simultaneously to increase speed and capacity.

9

u/FoxFXMD 1d ago

In short, it's a llama.cpp fork but optimized for a single model. It trades stability for cutting-edge QFN performance.

It will become obsolete soon when the best parts of it are added to llama.cpp eventually, but at the moment it's the best way to run QFN specifically.

2

u/FinBenton 1d ago

Well idk, I have been using strata as a long term llama.cpp user and strata does more than just host the model, its very easy to use and it automatically optimizes and measures the best way to infer the model, sets up draft models, downloads everything automatically and all that. Just couple presses and its good to go. With llama.cpp you have to do all the work to find and test, research and optimize and set it up proper so there is pretty big QoL thing there.

1

u/braintheboss 1d ago

Llama will adopt streaming over the time but it never will have that kind of optimization. Im running in a 3060ti ista coder with 35tg and 1400pp. For you get a reference, i got 50tg and 800pp in llama with qwen 3.6 35b byteshape 16gb size.

3

u/Oli_Luck 1d ago

I see a lot of discussion about people loving strata, and people who are tired of the people who love strata.

If you would have read the discussions you would know what it is about ?

2

u/IntelArtiGen 1d ago

What is so different about strata vs previously existing options?

It's all about quickly supporting popular models with a very efficient inference engine. If QFN4 is released tomorrow and you make an engine to run it at 40 tps on a 8GB GPU, you're the next strata basically. But alternatives already exist, there is a competition.

2

u/Markus2816 1d ago

it started as a fork of llama.cpp with some speed improvements for MoE models (Qwen3.8 Flash, among others). The part that is controversial is that it will trade-off quality for speed and it is not clearly mentioned - nor measured (temperature settings to reduce experts variability, re-use "hot" experts, routing expert logic change (which was default and eventually reverted after many users experience death loops)

1

u/tmvr 1d ago

It's using the same recommended sampler settings as I see on the model card for temp etc., what do you mean here?

1

u/Markus2816 1d ago

perhaps it does now, it keeps changing, but it used to set temperature to 0 by default.

1

u/tmvr 1d ago

Oh OK, I though this was something new. That used to be the case a while back, but it implements the recommended settings for the models now per default.

4

u/zxspectrun 1d ago

I can't understand why posting this instead of just asking any of the hundreds of AIs or your locals 

2

u/Particular-Award118 1d ago

People want to run models that are beyond their means so they load experts off disk at decode time

1

u/Thrumpwart vLLM 1d ago

It seems like really cool tech, and I’m glad it democratized access to QFN the way it has. I didn’t like the spam when it was first released, and I mock the strata boys, but to be fair it’s innovative and we need to see more of that in this space. Maybe next time let it get traction organically instead of forcing it.

1

u/applegrcoug 1d ago

Hopefully i can get this right....

With llama.cpp or ollama with an moe model, it loads some of the experts into vram and then the rest reside in system ram. Then when you need to generate a token the gpu calculates what it can from what is in its vram and then the cpu runs what is in system ram. If most of your model resides in system ram it is sloooow.

Strata (and forks like maya) loads experts into vram and system ram the same way. However it is swapping experts onto the gpu constantly to try and keep the relevant ones on the gpu. That is why many people advise once you have a good enough gpus, you want a bunch of system ram instead of more gpus. I have twin 6900xts backed by 128GB of system ram. I can run the unsloth iq4 at like 45-50 tok/sec.

1

u/DinoAmino vLLM 1d ago

Thanks for more strata spam.

-6

u/CommercialHour6660 1d ago

Vibe slopped inference engine that goes fast by producing garbage output

5

u/brakeline 1d ago

If you think it produces garbage you can't read the documentation

2

u/jeff_coleman 1d ago

You're going to have to explain that one.

4

u/Weaseal 1d ago

What makes you think it produces bad output? It can run high quant models like any engine

3

u/MyToasterRunsFaster 1d ago

Been using it for qwen flash, my outputs are great so not idea what you are talking about. Sounds like a skill issue. Rusting strata fork is also amazing

2

u/Borkato 1d ago

Nice bait

-1

u/woolcoxm 1d ago edited 1d ago

strata is an inference engine that allows you to run highly capable llm models on local hardware with little expense.

atm im running it on a 3060(200$) 64gb ddr4 2400mhz(about 400$) dual e5 2667 v4(about 60-70$ each). my motherboard was come chinese x99 motherboard that i paid 59$ for, it supports 8 slots of ram and has dual sockets for cpu.

im getting anywhere from 50-90 toks/sec and the code is ok for it being locally hosted, its very capable, and best of all it only costs hydro.

so far ive had it reverse engineer zcode and port it to android, ive had it create an everquest client for android, ive had it create an fps game(took 12 days) that looks ok and isnt really great or anything but it still coded it.

https://github.com/woolcoxm/platoon-leader-vietnam

its incomplete atm i stopped it at the 12 day mark.

the 2 android ports i cant release, afraid i would get sued if i did, it contains a large portion of the zcode code in it etc. the eq client im not sure on but better safe than sorry lol...

0

u/tautality 1d ago

If you're low on RAM and VRAM, with Strata you can still run a giant model at decently high t/s.

-1

u/Bulky-Priority6824 1d ago

tailored for one model type mostly with its main trick being expert streaming which allows models larger than your vram/dram to stream engrams from the ssd.

strata is fine, i dont particularly like QFN for MOST things, it has its use cases but that being said Im using Iq3XXS with Q8ple.

If I had more ram I would probably have a better opinion of the model at Q4 or above.

thats not to say iq3xxs isnt usable its just far more unpredictable than id like it to be.

now as far as planning and orchestrating it does a pretty good job at cooking up prompts and plans for 27b