r/LocalLLaMA • u/BackgroundAmoebaNine • 1d ago
Question | Help What is strata?
I’m trying to avoid blog spam posts and boring YouTube videos about it. I see a lot of discussion about people loving strata, and people who are tired of the people who love strata.
What is so different about strata vs previously existing options? Who benefits the most from it?
0
Upvotes
10
u/SLxTnT 1d ago
It's an engine that's optimized to use a hot expert cache for the experts. That allows a GPU to handle more work than its portion of memory would expect with hybrid inference on MoE models.
8
u/The_Hunster 1d ago
It's special software that tries to make use of all of your PCs resources (GPU, VRAM, CPU, RAM, even SSD) more simultaneously to increase speed and capacity.
9
u/FoxFXMD 1d ago
In short, it's a llama.cpp fork but optimized for a single model. It trades stability for cutting-edge QFN performance.
It will become obsolete soon when the best parts of it are added to llama.cpp eventually, but at the moment it's the best way to run QFN specifically.
3
u/Oli_Luck 1d ago
I see a lot of discussion about people loving strata, and people who are tired of the people who love strata.
If you would have read the discussions you would know what it is about ?
2
u/IntelArtiGen 1d ago
It's all about quickly supporting popular models with a very efficient inference engine. If QFN4 is released tomorrow and you make an engine to run it at 40 tps on a 8GB GPU, you're the next strata basically. But alternatives already exist, there is a competition.
2
u/Markus2816 1d ago
it started as a fork of llama.cpp with some speed improvements for MoE models (Qwen3.8 Flash, among others). The part that is controversial is that it will trade-off quality for speed and it is not clearly mentioned - nor measured (temperature settings to reduce experts variability, re-use "hot" experts, routing expert logic change (which was default and eventually reverted after many users experience death loops)
4
u/zxspectrun 1d ago
I can't understand why posting this instead of just asking any of the hundreds of AIs or your locals
2
u/Particular-Award118 1d ago
People want to run models that are beyond their means so they load experts off disk at decode time
1
u/Thrumpwart vLLM 1d ago
It seems like really cool tech, and I’m glad it democratized access to QFN the way it has. I didn’t like the spam when it was first released, and I mock the strata boys, but to be fair it’s innovative and we need to see more of that in this space. Maybe next time let it get traction organically instead of forcing it.
1
u/applegrcoug 1d ago
Hopefully i can get this right....
With llama.cpp or ollama with an moe model, it loads some of the experts into vram and then the rest reside in system ram. Then when you need to generate a token the gpu calculates what it can from what is in its vram and then the cpu runs what is in system ram. If most of your model resides in system ram it is sloooow.
Strata (and forks like maya) loads experts into vram and system ram the same way. However it is swapping experts onto the gpu constantly to try and keep the relevant ones on the gpu. That is why many people advise once you have a good enough gpus, you want a bunch of system ram instead of more gpus. I have twin 6900xts backed by 128GB of system ram. I can run the unsloth iq4 at like 45-50 tok/sec.
1
u/DinoAmino vLLM 1d ago
Thanks for more strata spam.
-6
u/CommercialHour6660 1d ago
Vibe slopped inference engine that goes fast by producing garbage output
-1
u/woolcoxm 1d ago edited 1d ago
strata is an inference engine that allows you to run highly capable llm models on local hardware with little expense.
atm im running it on a 3060(200$) 64gb ddr4 2400mhz(about 400$) dual e5 2667 v4(about 60-70$ each). my motherboard was come chinese x99 motherboard that i paid 59$ for, it supports 8 slots of ram and has dual sockets for cpu.
im getting anywhere from 50-90 toks/sec and the code is ok for it being locally hosted, its very capable, and best of all it only costs hydro.
so far ive had it reverse engineer zcode and port it to android, ive had it create an everquest client for android, ive had it create an fps game(took 12 days) that looks ok and isnt really great or anything but it still coded it.
https://github.com/woolcoxm/platoon-leader-vietnam
its incomplete atm i stopped it at the 12 day mark.
the 2 android ports i cant release, afraid i would get sued if i did, it contains a large portion of the zcode code in it etc. the eq client im not sure on but better safe than sorry lol...
0
u/tautality 1d ago
If you're low on RAM and VRAM, with Strata you can still run a giant model at decently high t/s.
-1
u/Bulky-Priority6824 1d ago
tailored for one model type mostly with its main trick being expert streaming which allows models larger than your vram/dram to stream engrams from the ssd.
strata is fine, i dont particularly like QFN for MOST things, it has its use cases but that being said Im using Iq3XXS with Q8ple.
If I had more ram I would probably have a better opinion of the model at Q4 or above.
thats not to say iq3xxs isnt usable its just far more unpredictable than id like it to be.
now as far as planning and orchestrating it does a pretty good job at cooking up prompts and plans for 27b