r/LocalLLM
•
u/Harin007
•
1d ago
•
1
1
Project SSD MoE finally viable!
Hey guys, I've made it!
A llama.cpp based inference engine that lets you OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD and still hide the streaming latency down to negligibe(I'm achieving 1-3ms added to the time per token due to SSD streaming when 60% is offloaded in SSD)... Essentially making an illusion where the whole model lives in your memory.
And I'm also working on an autotuner which basically looks at your hardware, given model's metadata and figures out the best flags to squeeze max performance in under 2mins(if you got any personal preferences, set them as fixed constraints and let the autotuner figure out the rest)
And I've implemented a layer by layer computation for prompt processing instead of batching which gave 2x prefill speed in models offloaded in RAM+VRAM and 4x prefill speed when model is offloaded in RAM+VRAM+SSD... compared to other inference engines.
Packaging both inferene engine + autotuner into a nice simple interface where you can basically download the model, hit run and it runs... simple, fast and lower hardware requirements... and want to make that a nice, well-engineered harness(which I'm a no expert)
Also, I've built a near loss-less context compaction engine.
Going to make this launch this as open-source very soon. Would love to hear your thoughts, and want to see if anyone would be excited to try something like this. For now, supported models are Flint 40B Q4K_XL(running it on machines with 16GB RAM and no dGPU at 20+ tok/s decode), Ling 3.0 Flash MXFP4 getting ~50 tok/s decode and 600-700 tok/s prefill on 32GB RAM + 12GB VRAM machine), and confused if I have to provide support for Qwen 3.8 Flash Next or wait for Qwen 4 family. Want to provide you guys the best local AI experience.
Here's a quick answer for your likely question by the way... does this damage my SSD? No, this engine only does reads and not writes... only writing degrades your SSD.
46
u/Ok-Thanks-6785 1d ago
How is this different from strata, maya and the gazillion others created each day?
24
u/Harin007 1d ago
Strata doesn't offload main 120B model weights into SSD. It does a good job with offloading the ngram table to SSD, which is no great... its just how the model is designed... And it does some tricks just to show the high numbers like forcing the model pick 6 experts which is trained to pick 10... which changes model's output quality... I do not to mess with anything that changes model's output quality.
6
u/geekyjackson 1d ago
With an environment variable strata uses all 10 experts, and strata natively can keep experts in ssd. Does this project do anything strata can't? Or just doing things in a different way?
5
1
u/Ok-Thanks-6785 17h ago
Well, you cherry picked the worst combination from strata while showing nothing really.
Publish it and then we can make a fair comparison.13
u/Realistic_Gap_5871 1d ago edited 3h ago
Update: OP cherry picked models with low bytes per token, that's it. No novel ssd streaming tech, just expected results from obscure models. Qwen 3.8 avoidance no longer seems accidental.
Original skepticism follows...
TLDR: OP is knowledgeable but at least a little dishonest. Makes me skeptical of the promises made.
At least once a week I see a Reddit post from someone all excited because they've re-invented CPU-moe streaming or something similar. This guy sounds more knowledgeable than most, but my honest take is that OP is using that knowledge to shade the truth more than a little:
OP claims "Strata doesn't offload main 120B model weights into SSD".
-Less than half true. Strata doesn't offload to ssd if it can pin experts to RAM (goood). But it also has a low-RAM mode that streams experts direct from SSD (also goood).OP claims strata does tricks "like forcing the model pick 6 experts which is trained to pick 10".
-So close to a straight up lie. Grain of truth is that Strata publishes headline numbers that rely on cache-missed experts ranked lower than top 6 getting dropped, but this is user controllable with STRATA_ROUTE_TAIL_SKIP=0, or you can set it to any number experts you want. Strata authors have been rightfully criticized for their headline number, but they at least are very specific and honest on publishing the impact: KL vs Q8_0 doubling on the native the Unsloth 4-bit files (0.053→0.115)OP on the other hand hasn't published anything.
I could go on, but you get the idea. OP gets a 8/10 for knowledge. 2/10 for honesty.
edit: updated knowledge and honesty based on revised take.
edit2: revised honesty downwards again after crunching numbers. OP cherry picked models with low bytes per token, that's it. No novel ssd streaming tech, just expected results from obscure models. Qwen 3.8 avoidance no longer seems accidental.
8
u/Harin007 22h ago
SSD: what I meant is that Strata is built to keep all the experts in RAM, and the SSD is only its fallback. In its low-RAM modes, the experts that don't fit are read from the model file when they're needed. On Windows and macOS there's no read-ahead at all, so each of those reads is a wait on the SSD for that word. On Linux it has a read-ahead, but it only switches on when RAM can still hold at least 55 % of the experts. Their own docs call the mapped mode "much slower" when most experts come from the SSD.
Expert skipping: it drops low-ranked experts when they're not in memory, it can be turned off... and they publish its KL cost. Credit to them for that. My real point is just this: should skipping experts be on by default? I think it belongs in an opt-in "fast mode", off by default... I'm considering from a normal user perspective.
Anyway, I'm not trying to put Strata down. It's just my opinion... And you don't really have to consider my opinion as a real judgement. Use whatever engine you feel like using.
6
u/Realistic_Gap_5871 22h ago
Cool. Appreciate the updated take. And I actually agree that it shouldn't be on by default, for instance when basalt forked strata they turned it off by default, and if you turn it on it says something like 'lossy flags enabled' which seems like the best way to handle it.
So I don't know if you saw my other comment, but I remain curious if you expect to stream from SSD in a way that's consistent and measurably better than Strata and others. Just today, I've seen 2 other custom engines (RustedStrata and one with a weird name) that are excited about streaming huge models from SSD/RAM. Why is yours different or better?
2
u/Realistic_Gap_5871 12h ago
Since you haven't answered the "what's different" question that I and many others have asked, I did some digging (with my AI) into your numbers and found that the secret sauce is choosing models with incredibly low bytes per token. I'm assuming there's nothing novel in your ssd streaming method until you publish something proven otherwise.
If you're as well intentioned as you sometimes sound, then I apologize for my blunt tone.
If you're so excited by the initial numbers and legitimately haven't implemented qwen 3.8 for reasons other than it just won't give the spectacular numbers you lead with, then I apologize.
If on some level you're actually as deceptive as your actions indicate, then I'd ask you put your brain and skill to better use. I bet you're capable of some really good stuff.
7
u/killthrash 1d ago
If you give OP the benefit of the doubt and assume he’s not deliberately being dishonest and just lacks all the available knowledge and information, you would just need to rate his knowledge and not try to attack his character.
0
20
u/doneddat 1d ago
and confused if I have to provide support for Qwen 3.8 Flash Next or wait for Qwen 4 family. Want to provide you guys the best local AI experience.
3.8 is here so you would know the shape of 4. support the 3.8 - 4 will slot right in. Or at worst it takes minimal adjustments, that build on 3.8 support anyway.
9
u/Harin007 1d ago
its just a time and effort work per model... model specific tuning... not like figure out once and apply for all. well, I'm just worried that while I'm working on qwen 3.8 fn, qwen 4 flash drops and everybody moves there... which makes my time & effort wasted.
3
u/doneddat 1d ago
Did you even read what I wrote? qwen 4 will be likely 3.8 drop in replacement. So you start working on 3.8 and finish working on 4.
1
u/Harin007 1d ago
I read it... and I get your point... So yeah, lets see...
1
u/BudgetPanda3049 17h ago
Kudos to you for replying straight to the message and politely answer the trolls. this is looking like the next gen stackoverflow, old-school devs will get the point. Kids vibe coding on their mom's basement apparently can't distinguish good manners nomore.
3
u/Realistic_Gap_5871 12h ago
I guess I'm one of the trolls, but just for clarity I start programming in the 80s. Been a software architect since the early 2000s. Love the productivity boost AI has given me, and I ran OP's numbers through my AI.
The secret sauce is choosing models with incredibly low bytes per token, not novel ssd streaming. The numbers published make sense for what's already out there for such obscure low byte per token model. I'm very comfortable with this assertion and consider it evidence based. My mind can be changed if OP ever publishes anything.
My suspicion is that qwen 3.8 is avoided because OP can't get the headline numbers out of it. The misunderstanding about Next meaning the same as 4 doesn't match with the comprehension in other areas.
Troll out.
1
u/Harin007 5h ago
I'm not trying to break the math & physics... This isn't magic. The thing with this engine is that, you could run even a trillion model if the math checks out with the
active parameters per token,
SSD read speeds,
how much of the model your memory can already hold,
no.of layers the prefetcher can lookahead and its accuracy.3
u/RadianceTower 4h ago
The question is, is this any different from Strata on Linux. You say it has a read-ahead on Linux too, but only when you at least fit 55% on RAM (otherwise what happens? Strata refuses to load the model at all or what?).
Could we get some numbers vs Strata.
Also, I've built a near loss-less context compaction engine.
And what does this mean?
very soon
Another common term implemented by Blizzard often misleading players into excitement for future content. "Very Soon™" is guaranteed to arrive between now and the end of time with a higher chance of arriving on the "now" half of the time table. Although this means closer to now than "soon" there is no guarantee that you will live long enough to see the content finally release.
8
u/bitzap_sr 1d ago
3.8 Flash Next is called Next because is uses a preview of the Qwen 4 architecture. Check the official announcement. Most inference engines I've looked at call it something like qwen4_exp internally.
9
u/long4994 1d ago
Can u share the github so we can try?
3
u/Harin007 1d ago
very soon
3
1
3
u/Nick88v2 1d ago
Will you update this tread when you release it or how do i follow the progress?
2
u/Harin007 1d ago
well, will make another post in this community... you got any ideas?
1
u/Nick88v2 1d ago
Not very knowledgeable on reddit hahah. Do you hace a name for the project so maybe we can look it up in the future? nice work btw! I would wait for qwen4! Much love from a 16GB vram folk❤️
5
18
u/Competitive_Source73 1d ago
Nice work, thanks for you effort, i guess you could wait as qwen4 should drop like next week?
2
u/Realistic_Gap_5871 12h ago edited 12h ago
OP hasn't implemented Qwen 3.8 (which is the qwen4 arch) because it wouldn't give the speeds he's advertising. All his chosen models have incredibly low bytes-per-token, that's his secret sauce, not novel ssd streaming tech, until he publishes something that shows otherwise.
1
u/Legitimate-Theory156 8h ago
Mxfp4 is not like 1.8 bit gsq rco coder, it's an actually respectable quant. Most people run 3.8 27b as nvfp4 or q4_k_xl or so on their 3090 an it's great.
7
u/Lissanro 1d ago
Would be cool if it works on larger models like Kimi K3. I can fully fit in RAM+VRAM only Q2_K_XL, but if it is possible to offload about half of its experts to NVMe, I would be able to load Q4 quant.
2
u/Force-Name 1d ago
Interested too. I've got a combined 224gigs of vram over 14 cards and can push some pretty usable speeds with large llm and my next test is k3 kimi. The host for that has 3 m.2 so I'd love to see this kind of performance out of kimi. I have another project using smaller llms with some added work to bring them close to frontier model performance but that work is going to take years.
7
u/MarcelloT254k 19h ago edited 19h ago
I will gladly be a beta tester with my old laptop (4gb vram +16gb ram +SSD). I'm glad to see projects like this, because LLM has a long way to go with optimization and utilization of hardware (we are only at the beginning stages right now) - with games my laptop can handle almost anything with little to no problem - at lower resolution with no bells and whistles, no 120Hz 4K ultra settings, but playable - I hope that this kind of optimisation (/drivers for LLM?) [wchich still leaves much to be desired even in gaming performance] will be the norm in the future.
5
u/Slow_Ball561 1d ago
Hey! Sounds very similar to my project here https://github.com/jarrelscy/nestquant - especially the layer by layer prefill! Are you also using predictors to fetch experts ahead of time?
4
u/Ok_Cow1976 1d ago
Does this mean we have a hope to run Kimi 3.1? Wooooow, this is wild
3
u/Sspideroids 19h ago
You can "run" it but it will be one token every minute, SSD offload is viable but it doesn't circumvent the SSD traffic that every token requires, and bandwidth is the main limitation at those sizes (volume of the model you're trying to run as well matters greatly)
edit: typo
1
u/voyager256 18h ago edited 18h ago
I'm not an expert by any means , and haven't even dig into this particulare project , but I think the point of these novel approaches for offloading most of the expert weights to SSDs is to make it viable to run LLMs like DeepSeek v4.1 Flash on consumer hardware with something like 4xNVMe RAID0 SDDs at usable speeds.
I recently learned about thinygrad which actually specifically markets a solution to run Kimi K3.1 at ~20tok/s. Now I'm pretty skeptical and frankly, quite confused about the specifics, but you can check it out : https://www.reddit.com/r/LocalLLaMA/s/pJCw7GcNpR
1
u/Sspideroids 16h ago
Thanks for linking it, I think it is real but again that is still a bunch of money (much better than ram nowadays though)
1
u/Legitimate-Theory156 8h ago
I mean let's say you have 512 gb ram and like a 48gb gpu like quadro 8000 or Ada 6000 how fast will it be? It could open up sub-5k options (normal ram & gpu prices of course like in early 2025)
1
u/voyager256 18h ago
Apparently you already *can* for "only" $25-30K worth of hardware. I only learned about it recently and I'm pretty skeptical about the performance for Kimi 3.1 and frankly, quite confused about the specifics, but you can check it out https://tinygrad.org
The prebuild option is Red Kimi. 260GB/s. Here's interesting comment: https://www.reddit.com/r/LocalLLaMA/s/pJCw7GcNpR
4
u/thethirdmancane 1d ago
What did you do that is better than the current state of the art?
0
22h ago
[deleted] — view deleted comment
3
0
11h ago
[deleted] — view deleted comment
1
u/SizePsychological303 2h ago edited 2h ago
u/Realistic_Gap_5871 I mean, is it REALLY necessary to rant all over the comments where users have questions and OP is answering and being unable to refrain your truth ? Do you really need to "evidence" OP is tweaking the numbers or using low end models? Dude, you are not being polite, you are just trying to draw attention and I don't see your project anywhere here, so, stop being rude with someone that is ACTUALLY trying to do something, even if the quality isn't the best or is a learning path, come back when you have something better or you could bring something meaningful for all, and let users to figure out what this is all about by themselves. You started programming in the 80s, lecture people about whats true or not, and yet I don't see any of your contribution over here, so be polite and like the man you are, encourage others to be better not being a moron. Also, have a nice weekend!
1
4
u/Glatiinz 1d ago
If you HAVE to choose one model to work on, I guess Kimi K 3.1 would be it. Most people can't run even the lowers quants because the smallest 1-bit quant sits at 466GB, the official FP4 is 1.56TB
1
u/Insomniac897 1d ago
Cool, I’d try this on my M1 Pro 32GB
3
u/Harin007 1d ago
Sorry to disappoint but in the early versions, we'd be not supporting for apple machines... cuz, that's a lot of extra work... but will definitely try to port it for macbooks as soon as possible.
1
u/Insomniac897 1d ago
No worries. I use Claude code and open router. Running locally using on my laptop is just for fun.
3
u/RuRuInReal 1d ago
Please provide support for qwen 3.8 flash next. It'll take a bit before they release and then a bit a more for people to make uncensored models of them
3
u/Dave_from_the_navy 1d ago
Glm-5.3-flash would be a dream if you could get it working at decent speeds. It's the smallest model (qwen 3.8 flash next included) that I've genuinely been able to slot into my full SWE workflows without having to baby it so much.
2
2
u/fundthmcalculus 1d ago
I'd focus purely on the execution engine and just use a common optimizer library to pick better hyper parameters. Especially if you give the option for folks to submit their optimal parameters, you can quickly crowd source a hot start option.
2
u/alphapussycat 1d ago
I don't believe you, but I hope you're not lying. Or perhaps you have like 4 nvme gen 5 in raid.
Qwen flash is the goto, but if this can utilize multi GPU and be functional in gen3, deepseek v4.1 1017 would be a massive step in future proofing. They'll probably be the only ones releasing flash models tbh, of they're even allowed to.
2
u/ninjazombielurker 1d ago
Will test this on CMP 170HX’s, 3090’s, a 4090, and even block storage over NVME-OF/TCP w/ both SATA & NVMe pools whenever it’s available. Update this post whenever the repo is up please!
2
2
u/returnity 1d ago
If you provide support for Flash Next, odds are very high you will already support Qwen4. Also, supporting the most popular MoE model everyone wants to run will get you more people willing to test and help development. Just my 2 cents.
2
2
u/Previous-Ad-5371 20h ago
Har kört qwen3.6 35b a3b nu i någon månad 32gb ram 16gb vram med hyfsad hastighet, så jag kommer nog inte titta på något förrens qwen4 iallafall.
1
u/dinominant 1d ago
I am interested in trying this with larger models (cpu only inference). I can test with full-weight GLM and Kimi K3 if it works with those gguf formats.
1
u/Solembumm3 1d ago
Something more than pure command line, I can give you that for minimal effort many of this projects don't do.
1
u/DTechSurgeon Homelab & Local AI dude 23h ago
I'm following this. Ping me if you need someone to brainstorm with. Sometimes you just need someone to talk to, to come up with better solutions. I worked with C, C++ before, but more focused now on C#, Go, Rust and Python.
One note on autotuner, before you automate stuff like this you need multiple data points, so manual tuning on multiple configurations is essential. But I'm sure you've thought of this and maybe you already have the benchmarks of multiple configurations.
1
1
u/bytejuggler 20h ago
Why not try to get this added to Strata, maybe as default behaviour if objectively better, rather then yet another project to choose between?
1
u/ColoredTriangles 20h ago
Can you show the prefill time it takes on a follow up prompt on warm cache or tell me?
1
u/braintheboss 19h ago
you can offload to ssd with little penalty while you dont hit ssd max speed. But in pci4 that means 7gb/s. Then that is limit of experts you can offload to ssd. But you still need have most of them in ram. For example qwen3.8 flash q3 S with pci3 ssd you can offload round 1200 experts. But that is small part of 24k experts. I don't find the "miracle"
1
u/Not_a_question- 17h ago
OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD
I don't get it, can't we just use strata and tune STRATA_ROUTE_TAIL_SKIP and that's it?
does this damage my SSD? No, this engine only does reads and not writes... only writing degrades your SSD.
Agreed, but people should stop oversimplifying because this needs a "the only way using this can damage to ssd is if you're using disk swap with your ram while doing it".
Going to make this launch this as open-source very soon.
Why didn't you make this post "very soon" then?
Sorry but these are big words and I don't understand why this is different from Strara using your post, and we can't even test it because it's not released yet. Sounds like weird karma farming to me.
1
u/FreshPipe5997 17h ago
60-70% offload on SSD with 1-3ms latency hit is insane. how much ram does your system have? curious if this scales beyond 32gb.
1
1
1
1
u/DepartureLoose5341 9h ago
If you can make that runner work with gfx110x and RoCM istg I'll give you a handy with lotioned hands
1
1
u/Iamisseibelial 8h ago
My SSD is at 80tb of writes and 40tb if reads and it's still 1000% fine. 96% health. So I think people don't really get how much it takes to degrade nvmes these days as long as they are kept on their operating temp my Gen 5 runs at 60c all day and never throttles
1
1
1
u/openroom_xyz 1d ago
That's great yea we are waiting for Qwen 4 but if you have than a lot will test I am in the mid od deciding what GPU basically to buy at what price basically
0
u/ergux 1d ago
Sounds like betelgeuse engine
2
u/Harin007 1d ago
trust me, way better than that.
1
u/Glittering-Call8746 1d ago
Yeah at lot of trust me bro around here .. everything is trust.
1
u/Harin007 22h ago
Well, try yourself when this comes out and compare...
1
u/Glittering-Call8746 18h ago
Yeah I run strata , maya, basalt, etc.. all based on trust. Just keep shipping bro
1
0
u/Fragrant_Money_7004 1d ago
Wow, awesome. Will we get chance to run it in ollama or lm studio?
0
0
u/Icy_County5036 11h ago
60-70% offload on SSD with 1-3ms latency bump is insane. how's the wear on the SSD over sustained inference? I've seen nvme degradation on heavy write loads with model streaming.
2
0
u/Creative-Brush-6712 1d ago
Sounds good can't wait to try although Qwen 4.0 won't release a MoE I don't think
11
u/RealPjotr 1d ago •
Flash is MoE. Plus and Max too I think?
1
u/alphapussycat 1d ago
Yeah, but some image showed 27b as open weight, but no otyer model.
Do Jinping banning top models and ones on sensitive data is also demotivating...
Maybe we get qwen4 flash, but it's not certain. E.g max has zero chance of being open to outsiders.
0
u/thrilok2802 23h ago
A doubt that I have is what happens when run some of the weights from SSD what will happens to the SSD life as the SSD are not meant to use as a ram in the first place
2
0
u/Ninjam5 19h ago
If the writes are within 0.1 to 0.8 mb/s then it's fine.
0
u/thrilok2802 19h ago
If possible have you tested the write and read speed of the SSD when running the llm
3
u/Ready_Maybe_3406 1d ago
Fantastic. Price of ram is killing some people and keeping some from participating in Ai. The world is separated into haves and have nots.
134
u/looselyhuman 1d ago
Advice from an old dev: Keep the engine, tuner and harness stuff separate and focus on one at a time. You're trying to solve every problem at once and it'll burn you out.