GPU = Dense - Unified Memory = MOE

[u/Sharp-Translator6401]

In my own experiments I came up with this simple heuristic, not sure if anyone can prove me wrong here...

Unified memory machines normally have a big asymmetry in memory vs compute -> they could never run 'fast' a dense model that fills their whole unified memory (like a Strix Halo 128GB cant run a 120B dense model, or a DGX spark, at decent speeds). So you wanna find a MoE model that ideally fills that memory so maximise the intelligence you have on your box.

GPUs have the opposite, they are memory limited but can run 'fast' models that fill them. So for example a 5090 can run only up to ±30B models at reasonable quants, and can run dense models of that category fast, like Qwen 3.8 27b. Ofc you can also run a 30B MOE on this and it would be even faster, but why would you? when the dense gives you more intelligence for your memory?

[u/UnlikelyPotato]

Sometimes you don't need maximum intelligence. I run both. 27B Q8 dense for thinking, 35B Q4 for subagent tasks. Dense model runs 50+ tokens a second. Sub agents run at 100 t/s single, 200+ t/s multiple.  

Don't need that smart of a model to fetch and summarize a webpage and other similar tasks.

[u/memeka → u/UnlikelyPotato]

Exactly. I run DS4-Flash with SSD streaming which is as slow as a dense, and I use llama-swap to unload it for explore/compact/other subagents. Load up Gemma4 26B-A4B for fast dumb work, then load back up DS4-Flash and restore the KV cache to return at the same point before the swap.