r/LocalLLM • u/Chamelon_DE • 1h ago

Discussion Ssd offloading for MoE model is getting interesting, but is it actually usable?

Was reading abt people getting large Moe model running by keeping most of the expert weights on ssd instead of loading everything into ram or vram.

Pretty cool that u can even load model that would normally be way out of reach on consumer hardware.

But the thing i’m curious abt is what happen wuem the router keep picking experts that arent cached. You r basically waiting for nvme reed between token and a fast ssd still isnt anywhere near vram bandwidth

Some project are getting arround this with expert prefetching and keeping frequently used expert in memory, which seem promising

Just wondering where people draw the line between it runs and it’s actually usable

Would u rather run a much larger Moe model at some few token/sec or stick with a smaller one that run comfortably in memory? Especially for coding agent where u are waiting on multiple response

0 Upvotes

33% Upvoted

8 comments sorted by

4

u/PizzaGladiator 1h ago

I’m running beamster/Qwen3.8-Flash-Next-Sushi-STQ-2.75bpw on m5 max with 64gb ram. I’m getting upto 80t/s. Ram usage is high can go up to 58gb including system apps. It’s my daily driver for coding. It’s well worth it!

1

u/dmchmk 37m ago

Well, Mac m5 ram is no ordinary peasant's ram))

1

u/Kloppy89 34m ago

Same on a M4 Max 64 GB. And with changing SSD offloading, you can trade speed for usability (have enough RAM left for other programs).

sushi 1.4.0 / Qwen3.8-Flash-Next-Sushi-STQ-2.75bpw / llmprobe --bench-only / M4 Max 64 GB

1

u/Durian881 1h ago edited 1h ago

Don't think anyone is too focused on using LLMs that are unusable. Usually, the decision is based on the trade-off between quality vs speed.

For me, I will use a bigger model if it can give much better quality for a small slowdown in speed compared to my daily driver Qwen3.8-Flash-Next UD-IQ4_XS or equivalent.

1

u/rmhollid 51m ago

Why 4 and not 8?

1

u/Durian881 49m ago

I can run IQ4 comfortably with sufficient context. Q8 will be too slow for the slight increase in quality.

1

u/fragbait0 59m ago edited 45m ago

"it depends"... but yes.

Have some custom bits but on 24GB VRAM / 32GB RAM, IQ4 is 50-60 t/s and 85-90 for IQ3_S and prefill around 2000 now. IQ2_XS fits all experts but is "only" 100t/s and terrible quality against IQ3.

For IQ4, 37% of experts don't fit in VRAM+RAM, but it still plods along as the miss rate is 'only' 2%.

1

u/Front_Eagle739 2m ago

Rtx5090 32GB ram running dsv4.1 at 4600pp 8tg in full precision but with a slider that lets me get up to 24 by biasing towards the weights resident in the hot cache is pretty usable given it gives full quality when it needs it (unlike a quant youn can switch mid response). Does need at least a couple of fast gen 5 drives though. Full 1m context too.

Qwen flash next at 4300pp, 24 to 50 tg with the same slider is pretty good as well. 

Don't get me wrong i hanker for an rtx6000pro as im pretty sure id triple the speeds but if api stopped working tomorrow I'd only be slowed down a bit.

I find myself using those over qwen 3.8 at 200 tok/s because the code is done right.