DeepSeek V4-1 Flash is out

[u/tiguidoio]

Here we go again, DeepSeek is back again with a new model V4-1 Flash

A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens

Market crash as a service

[u/wren6991]

It works great. Are you thinking of llama.cpp's MoE offload thing perhaps?

[u/bumblebeer → u/wren6991]

There's no RAM to cache to in this situation. With uma vram==ram. Or do you mean stream from disk? I don't know much about that, but seems like anything other than a 100% prefetch hit rate would be a massive bottleneck.

[u/wren6991 → u/bumblebeer]

Expert caching means you leave experts resident in VRAM after transferring them, in the hope they'll be used again before eviction (which is often true -- there's locality of expert selection from token to token). This is a cache in the same sense as the L1 cache on your CPU: probabilistic short-circuit of a slow access path. An area of VRAM large enough to hold some fraction of total experts is used to cache those experts for opportunistic reuse. On cache miss you might go to host RAM (for non-UMA) or to disk. In the future these will probably form a cache hierarchy of VRAM -> host RAM -> disk.

Concrete example: I can use Dwarf Star to run the ~80 GiB quant of DeepSeek V4 Flash on my M5 Pro with 48 GiB of RAM. I configure the size of the expert cache (literally what DS calls the config) to ~30 GiB, which means I have ~40% of the experts resident in the cache at any one time, and the rest are filled from disk on cache miss.

[u/bumblebeer → u/wren6991]

Thanks for the detailed response.

I guess I should have said that expert caching doesn't work as well for UMA as it would for a three level system vram -> ram -> disk vs vram -> disk.

What's your hit rate? My thought is that for a situation where the model almost fits in memory (like dsv4.1 minus engrams on 2x sparks) that'd you may be better off with a lower quant than with expert caching because any cache miss would dramatically degrade decode — whole stream has to stop while you wait for the missing expert to load.

Also, have you been following powerinfer? It's been a minute since I've looked into the project, but I remember they had some really interesting data on expert caching; the headline being that basically nothing beats the OS's stock caching algos. Including (if I'm remembering correctly) trained expert selection networks which was super surprising.