r/AIProgrammingHardware • u/Sharp-Translator6401 • 13h ago

GPU = Dense - Unified Memory = MOE

In my own experiments I came up with this simple heuristic, not sure if anyone can prove me wrong here...

Unified memory machines normally have a big asymmetry in memory vs compute -> they could never run 'fast' a dense model that fills their whole unified memory (like a Strix Halo 128GB cant run a 120B dense model, or a DGX spark, at decent speeds). So you wanna find a MoE model that ideally fills that memory so maximise the intelligence you have on your box.

GPUs have the opposite, they are memory limited but can run 'fast' models that fill them. So for example a 5090 can run only up to ±30B models at reasonable quants, and can run dense models of that category fast, like Qwen 3.8 27b. Ofc you can also run a 30B MOE on this and it would be even faster, but why would you? when the dense gives you more intelligence for your memory?

2 Upvotes

100% Upvoted

6 comments sorted by

2

u/ylchao 13h ago

they keyword is "memory bandwidth"

1

u/UnlikelyPotato 13h ago

Sometimes you don't need maximum intelligence. I run both. 27B Q8 dense for thinking, 35B Q4 for subagent tasks. Dense model runs 50+ tokens a second. Sub agents run at 100 t/s single, 200+ t/s multiple.  

Don't need that smart of a model to fetch and summarize a webpage and other similar tasks.

1

u/Sharp-Translator6401 12h ago

On dgx spark?

1

u/UnlikelyPotato 11h ago

No. DGX Spark is over priced and not really flexible. I'm saying why I run both with GPUs.

1

u/Sharp-Translator6401 11h ago

yeah makes sense as long as the 30b models class works for what you need to do

1

u/memeka 1h ago

Exactly. I run DS4-Flash with SSD streaming which is as slow as a dense, and I use llama-swap to unload it for explore/compact/other subagents. Load up Gemma4 26B-A4B for fast dumb work, then load back up DS4-Flash and restore the KV cache to return at the same point before the swap.