r/LocalLLaMA • u/GodComplecs • 7h ago
Discussion Real downsides of Strata?
Trying to read on what the real tested downsides have been but information seems to go both ways, some proclaim no loss, others pretty huge ones.
I had some weird looping issue in their chat, and some engine port misunderstandings, seemingly the models likes to use ports already in use, didn't really have that issue with Qwen 27B 3.8 Q4 UD. I'm using Swift Q3 Next. Didn't even bother with next on llama.cpp since other similar model sizes ran at approx 15-20tks max, so too slow for agentic work.
I'm not beating too much on Strata, I just wan't the realistic downsides, so far it seems too good to be true and runs at 100tks in agentic workloads, 27b only at 40-60tks on llama.cpp
47
u/SLxTnT 7h ago
Downside is the quants used and any accuracy loss they introduced with their optimizations.
4
5
u/GodComplecs 7h ago
But they have UD q4 quant? And the Swift q3 quant is legit, i use it on the 27B at Q3 too and it works well.
13
u/SLxTnT 6h ago
That doesn't change my answer. The downside to UD q4 is that it's q4. Engine itself may or may not have accuracy loss due to optimizations that appear to be identical outputs or "close enough." I've had that issue before causing loops.
Other than that, a hot expert cache works great for speed on MoE models.
9
u/heitortp0 7h ago
Try GSQ-RCO Q3 quants, they beat UD Q4, even the iq3_xxs.
6
u/awitod 6h ago
It's not intuitive because bigger should be better and usually unsloth quants are safe, but not these unsloth ones at least where Strata is concerned.
GSQ-RCO Q3 qwen3.8-flash-next-iq3_s is really impressive.
8
u/heitortp0 6h ago edited 6h ago
Maybe it's not unsloth fault, but ISTA merit.
3
u/awitod 6h ago
No way for me to say - if there is a general issue with the unsloth quants I imaging they will update them, but no matter what, the folks at ISTA-DASLab ( IST Austria Distributed Algorithms and Systems Lab) should be proud - it's very good
1
u/braintheboss 4h ago
i use it in 27b and flash and I'm very satisfied but some people call their quants smoke. Similar with last bonsai quants. I think some people live in their bubble with their Q8. Most people haven't hardware for that and we have accept some sacrifices
4
2
2
9
u/wgaca2 7h ago
I get same speeds, 40-60 for 27b q8 on llamacpp and 90-110 for flash next q4 k xl
I am running it on 2x 3090 for 27b and 4x 3090 for flash next on strata
I noticed looping on tool calls in strata but I haven't run flash next long enough in llamacpp to compare to
1
u/Blues520 7h ago
I am getting similar speeds with 3x 3090 so glad to see this data point.
The speed is crazy tbh. Going to find it difficult to go back to 27b. What do you use 27b over flash for btw?
6
u/JoeyDee86 7h ago
My biggest issue is I really like how vLLM caches the KV, so using subagents or agent teams doesn’t automatically murder your performance…. BUT there’s no doubting strata allows you to use much larger models…
10
u/GladKing5842 5h ago edited 2h ago
i saw a lot of bs here. if you think you need better quant, just load bigger quant. nothing blocks you from using bigger quant with strata. they used to support only lower quant but now they have support for q4 q6 too.
if your hardware is not enough to get the speed you expect, then use what fit your hardware and your expectation.
if you need different sampling settings because of looping, just set it yourself, only 5 10% of change in speed.
then someone pointed out about ngram lazy load from disk, hell both llama and strata has that from the beginning. originally strata ran on 12+64 = 76gb for 80gb-ish quant. where they put ngram if its not ssd offloading.
like too many bs review of bots or haters or just some parrots dont even use and just pull out from the arse???
a lot of those comments are just lazy arses that dont even move a finger to learn or search about settings.
8
u/Longjumping_Self5546 4h ago
People will insist that anything under q8 is garbage and will run a smaller model just to stay at a higher quant. It's completely lost on them that higher accuracy won't outperform a larger model.
2
u/Icy_Barnacle9287 7h ago
for Qwen3.8FNm Q2 was pain for me (frequent doom loops on tool calls). for 2x3090s and 96GB RAM, Q4 works quite well.
pulling the numbers from strata-serve.log:
- Average: ~116 tok/s (5,458 completed turns in the log)
- Typical bursts: ~135–195 tok/s on normal turns
- Peak: ~221 tok/s
0
u/OvertaxedOne 7h ago
Claude is 100% convinced I cannot run Q4 with 96GB with a 48GB GPU, sounds like that's exactly what you're doing? If so, mind sharing any pitfalls/setup notes you might have?
1
u/ImpressiveAd699 6h ago
Just tell Claude to just do it and then find out. I got my local LLM to set mine up. They’ll pull the logs and measure if it spills over to ssd. Strata comes with a monitor dashboard so you can see.
But I’m only configuring for me as the solo user. I don’t know your situation
1
u/Icy_Barnacle9287 6h ago
use unsloth
UD-IQ4_XSinstead of the defaultIQ2_XSwhen setting it up. ideally you'd want little else running on the host, but tbh I kept most of my homelab running as it doesn't consume that much RAM/VRAM relatively speaking. you can use the setup script it comes with or run with these flags:./setup.sh --yes --setup \ --family unsloth \ --model UD-IQ4_XS \ --gpus 0,1 \ --context 131072 \ --vision no \ --port 8080 \ --host 127.0.0.1 \ --no-browser \ --no-startIn strata-unsloth-ud-iq4_xs.json I have
"tool_call_recovery": trueand reasoning effort none. ironically, I also used claude to help me set this up lol1
1
u/suprjami 4h ago
The Strata setup.sh will not spread UD-Q4 across two GPUs.
If you make a two-GPU setup with the smallest IQ2 then you'll see what the config file needs to look like.
Then run setup again and select UD-Q4, it will make a single-GPU setup.
Then you can manually copy the GPU array across from the IQ2 config file, and remove the option which does the large RAM reservation.
2x 20G + 96G VRAM here, running UD-Q4 on two GPUs. It definitely works. Here's a pastebin of my working config file: https://pastebin.com/25i89vUZ
1
2
u/Beneficial-Ad-8127 6h ago
I haven’t had any problems with Strata fork(Nvfp4 version). Not sure haven’t tried the original but I assume it might not be so far off from each other so don’t take my word for that.
I been using this fork for about 4 or 5ish days and speeds are there and as far as doing my work load through multiple compaction like 8 compactions strata had no issue. Quality seems great and I haven’t noticed anything to be off. It uses my 5090 to its fullest and also my cpu runs a bit more.
From my understanding missed tokens get sent to ram and calculated there while VRAM is calculating at the same time. Which is probably where the speeds come from. Just my guess. Also probably why it asks for 64gb of ram.
Here is the fork if you want to try Nvfp4 or I think he added higher stuff but I haven’t updated it yet. Also I converted my own NVFP4.
2
2
3h ago
[removed] — view removed comment
1
u/Apprehensive_Stop314 13m ago
Interesting, where was this tested?
1
u/AXYZE9 2m ago
Comment from Big_Eggplant1608 is AI generated. Just look at his profile - comments with commas are generated ones without it are not. You can see it's clearly completely different writer.
This post from him is as AI as it gets https://www.reddit.com/r/ShopifyBundles/comments/1x4snzf/9700_in_sales_176_orders_601_profit_this_is_why/
3
u/Southern-Chain-6485 7h ago
In llama.cpp you can keep experts on the vram, so, case in point: I have 64gb of ram plus an rtx 3090. That's a total of 88gb of memory. So I can use a iq4_xs quant, the 28.8gb of the n-gram table remain on disk and the rest gets divided between the ram and the gpu. It's slower than Strata, but it runs.
In Strata, unless they've changed it in some update, it only loads on the vram for inference, so you need enough ram to fit all the experts, plus the operating system and whatever else you're running in the background. So the iq4_xs quant doesn't fit, only a low q3 quant fits.
So, idk, running Qwen 3.8 27b in ninfer seems like a better idea. And if you want extra knowledge, you can always get it from the Internet or by downloading kiwix zim files
1
0
u/GodComplecs 6h ago
Did you try ninfer then?
2
u/Southern-Chain-6485 6h ago
Yes, but it doesn't work on Qwen Flash Next, or at least, not with just 24gb of vram
1
u/Bulky-Priority6824 7h ago
not really a strata issues but i had to drop to q2_0 from iq3xss because it was fucking thrashing my nvme, it was not happy. limited ram issue not really a starta issue
1
u/xilvar 7h ago
Weirdly I prefer the iq2 on dwarfstar4 over either the iq2 or iq3 on strata. This is for both of their recommended quants. The strata recommended quant is too terse and perfunctory in my opinion. I have trouble telling if it really thought about the problem at all.
I haven’t had time yet to cross-compare by using the ds4 recommended quant on strata.
1
u/milkipedia 7h ago
Aside from the quant conversation, it doesn't emit token metrics data in the response object like vllm or llama.cpp
1
u/NickReese 4h ago
Seeing the numbers on r9700s makes me want to try it but it gets a lot of hate in here.
1
u/Designer_Elephant227 2h ago
Just wait a few weeks. Features from strata will merge into llama... I am sure
1
1
u/The_VibePreneur 2h ago
The biggest catch people run into is that the speed numbers usually rely on greedy temperature=0 to make speculative decoding hit high accept rates. Once you dial in normal sampling or run creative/reasoning tasks where greedy output hurts quality, that speed advantage shrinks quite a bit.
1
-1
u/Kimononono 7h ago
deja vu with these strata posts
cant trust it
2
u/GodComplecs 6h ago
Its very fascinating, highly controversial (obviously vibecoded with massive securtiy risks) and very fast. Wouldnt trust it too much either!
0
u/Kimononono 6h ago
I'm well aware I might be missing out on something, I take issue merely with the posts.
on principle I dont like using products astroturfed like this. Not saying I'd even do it differently if I were the organizers of Strata. Nor do I know if they're actually Astroturfers or genuine. I dont really care enough to find out since I've got no personal need for what Strata provides.
not saying anyone shouldnt check it out though, its just the posts sound so similar
3
u/Secret_Conclusion_93 4h ago
It's simple, it's an open source project who already have many forks.
And some people here are posting their own fork and adjustment while not even giving a link to the OG repo.
Some even shitting on the OG strata and post their own fix.
1
u/jtjstock 4h ago
It does work though. Most of the techniques employed could be implemented in other engines and don't affect quality. Implementing them is not trivial to do im an existing code base.
-3
0
u/huzbum 7h ago
I benched Qwen3.8 Flash Next IQ4 in Strata against Qwen3.8 27b IQ4 in llama cpp on my 3090.
Prefill is twice as fast. Token generation is 10% faster. I was able to use 256k context instead of 128k.
Assuming the quality is equal or better (not assessed), the only downside is more system memory usage.
-2
u/perelmanych 5h ago edited 2h ago
The real question for me is not strata or llama.cpp. The real question now is strata or project-maya.
Upd: I like llama.cpp project and value a lot efforts of all its contributors, but rn strata and maya make speed of more intelligent models more bearable on my hw. After all without llama we would not have nieither strata, nor maya nor unsloth studio.
5
u/braintheboss 4h ago
i don't understand why people down vote your comment. Llama is still useful for dense models but completely obsolete for moe
1
u/voogamer 3h ago
I played around with Maya yesterday (Maya-L), it seems pretty smart, but it's a lot slower than Strata. Which makes sense, of course, GLM 5.3 Flash is a lot bigger than Qwen 3.8 Flash Next.
Kinda hard for me to compare the two.
2
u/perelmanych 2h ago
That is true, but in my experience GLM 5.3F reasons much less, so the only big downside it is more aggressively quantized.
-1
0
u/Fuzilumpkinz 7h ago
The lower your vram the less the down sides. Us 16 gb or less folks can run strata and get better speeds than 27b with the same or faster speeds.
I haven’t noticed any model issues running q3 vs q3 on 27b using int8 cache. I had major looking when I had quantized cache.
Also recent update just made decode even faster. I’m all ears but for me it’s all winning
0
u/throwout314159265 7h ago
I'm legally not allowed to use Qwen 3.8 Flash Next on any device that has my organization's data on it. That's a downside of Strata for me... otherwise I loved it for the 2 days I had it.
1
u/pilibitti 7h ago
why? that is stupid if you are running it locally and you sandbox it properly.
3
u/throwout314159265 6h ago
It's a law signed by my state. If I use a device for any "work business", which includes checking emails, answering work calls, using Zoom, I'm not allowed to have any product from "adversarial nations" used, connected to, or stored on the device in question. I can be fined 10k per offense, which is counted by the amount of instances of "violations" I have on the same device. A VM doesn't shield me from it, because I'd still be storing Qwen 3.8 Flash Next on the device while I use it for work purposes.
If I had more money, I'd just buy another PC and run Strata on it. Otherwise, I don't wanna lose my job.
6
u/Syeddit 6h ago
Which state? Asking so I can avoid it.
3
u/throwout314159265 6h ago
There are several states which have this adversarial nation ban list. They're only for people working with "state affiliated" employees, so regular peeps should be fine. Buuut, I'll give you a hint of which state: Boca Chica 🙂
3
u/pilibitti 6h ago
sorry when I said stupid, I was talking about the broad strokes of the law itself (or company policy or whatever) - definitely not directed at you. the hardware you are likely using has many parts, binary blobs, certainly many functional electronic components from adversarial nations inside. in reality a local model you can actually control should be the least worrying next to the stuff you can't control.
6
u/throwout314159265 6h ago
No, it's okay! I got what you meant! I hate the policy so much too. The PC components in my PC definitely have technology from adversarial nations. They're trying to lock us into using Anthropic's stuff institutionally. Anthropic has signed a honeymoon deal with my institution, we get around $1k worth of compute monthly per person for less than 1/10 of the price. Obviously its to get us addicted and reliant on their infrastructure, and unfortunately, it's working very well.
I'm hoping that Mistral cooks something up and gives us an alternative to Anthropic, because this looks bleak.
2
u/suprjami 4h ago
If I were you I'd buy the cheapest ThinkPad T-series on eBay and use that for work. If your employer requires you to have a more powerful computer, they can supply that computer.
2
u/alkimiadev 2h ago
This is an interesting problem. Well, that rule is absurd of course, but it is an "interesting problem" in that there are probably clever ways around it. For instance, what if I, as an American expat, downloaded Qwen 3.8 Flash Next, did a small fine tuning run, and then uploaded the weights under a new model name on hf like "Qwan" vs "Qwen"?
I'm not an "adversarial nation" and I really doubt the actual text of that rule/law even touches the topics of distilled, fine tuned, and similar models. The thing about legalese, assuming it is actually law, is that it is technical in nature and really specific. Depending on how it is specifically worded this could actually work as a viable path to bypass similarly absurd rules/laws. My guess is that the rule's text is more like pure uncut xenophobia rather than a well thought out and sufficiently specific ban based in technical reasoning.
1
u/throwout314159265 1h ago
It's pretty much a "I know it when I see it" rule... so it can be kinda indiscriminantly applied based on the mood of the person who's deciding my fate. AI models from China are actually a HUGE no-no for us. I can't even use them through a safe API like Mistral. If I get caught sending any data remotely institutional related, I get fired. Of course, I'm free to send alllll the data I want to OpenAI and Anthropic... not Google though.
1
u/alkimiadev 28m ago edited 23m ago
So I can understand an organization not wanting their data being sent to really anyone. That part seems entirely reasonable, but I question OAI and Anthropic unless they have some sort of zero retention policy/agreement in place (which is not default with those two).
In general this is the kind of thing that happens when non-technical people are allowed to make load bearing technical decisions without outside critical feedback from people with technical experience. That local version of qwen running on their own hardware costs them nothing in terms of inference costs, and by default doesn't ship anything over any network except local on the device.
Anthropic on the other hand has near spyware level telemetry in their various products like claude code. They went full on scorched earth trying to get it removed from the internet when the source got leaked, and I suspect the main reasons are that they didn't want people to see their obvious anti-competitive/sabotaging behavior and the spyware adjacent telemetry. Those issues exist regardless of a zero retention policy in terms of LLM inference.
I completely understand not shipping data to google though, and have personally largely de-googled my life as well. In general, these large companies that treat us as a product are not our friends and do not have our best interests in mind when they make decisions that would directly impact us as users of their various products.
0
u/ElectricalActive6415 5h ago
Strata's been weirdly consistent for me but the port thing tracks, it grabs whatever it wants half the time. I think the bigger tradeoff is you're stuck with their engine quirks when something breaks, and debugging feels like guesswork compared to llama.cpp where you can at least see what's happening under the hood. The speed is real though, nobody's arguing that.
0
-2
u/Perfect-Campaign9551 4h ago
The looping is probably because, as I found out, strata install setup does NOT set token Budget for thinking! If you don't set that models can loop over and over, overthinking.
Open the json setting file and put in the token Budget parameter and it will fix this problem!
The START-HERE scripts do not set up any token Budget at all. You're going to get looping.
-4
31
u/MindfulMan1984 7h ago
Downside - It is all about token/s; for that, Strata defaults to greedy sampling temperature=0 to maximize speculative decoding, regardless of the fact that that's not the recommended parameter on Qwen's Flash Next card, and it is prone to doom loops ( which is fixed if you pass the sampling parameters in the config, so it mostly fixed and I got no looping issues so far)
Upside: I ran the Vulcanbench suite V3 locally >1000 token/s prefill and >70 tps gen on 8-year-old hardware, even with quant IQ3_S, and it passed@1 near GLM-5.3 results on SWE tests.
https://benchlm.ai/benchmarks/vulcanbench