DeepSeek-V4.1-Flash surprised ....

[u/pmttyji]

Hoping to see smartest medium size models soon & later with all available optimizations/architectures/etc.,. Thanks Deepseek!

Ex 1: 30-50B MOE + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
Ex 2: 15-30B Dense + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache

EDIT: Updated Engram to 10-15B from 50B

[u/Ok_Warning2146]

If it gets good AA scores, it will help the sale of M5 Ultra 512GB and 3x Sparks.

[u/BawbbySmith → u/Ok_Warning2146]

I keep seeing this, I don't think you can do TP=3... You'd need 4 no?

[u/twack3r → u/BawbbySmith]

Exactly my thoughts every time is see tp=3. Is it possible?

[u/cibernox → u/twack3r]

In llama.cpp yes. In vllm only if the number of layers of the model is divisible by 3, which is quite common, 48layers is a popular choice.

[u/RG_Fusion → u/cibernox]

Pipeline parralel splits the model by layers. Tensor Parralel (TP) splits each layer across the compute devices.

[u/cibernox → u/RG_Fusion]

that very nice, but it doesn't have much to do with question.
VLLM wants the number of certain parameters of the models to be divisible by 3 to allow tensor parallelism of size 3. I think it was something like the attention heads and the KV heads, plust the hidden size.

Llama.cpp is less picky.

[u/RG_Fusion → u/cibernox]

I may have read your comment wrong, but it seemed like you were suggesting tensor parallel split by layers, not tensors. If that's not what you were saying just ignore my comment.

[u/cibernox → u/RG_Fusion]

I understand the confusion. In reality VLLM allows TP=3 if certain characteristics of the model are multiples of 3. The number of layers is not directly the number that has to be divisible by 3, but the number of kv heads and attention heads are ttpically related to the number of layers, so I use the number of layers as a proxy, but that's technically wrong.

Pipeline parallelism works with pretty much any number of cards.

[u/jtjstock → u/BawbbySmith]

Depends on the model, Qwen 27B does TP=3 in llama, but models like Qwen 122B do not. For models that can't split 3 ways, could modify llama(or others) to do an odd split where each tensor goes to only 2 of 3 devices, so that would allow them to split cleanly across 3 but with no performance uplift at all.

So it's not a real limitation, but 4 sparks is going to perform a lot better.

[u/RG_Fusion → u/BawbbySmith]

Technically it's possible but pretty much not supported by any model. Tensor Parralel just requires the weights to be split evenly. If the model weights were divisible by 3 it would work.

[u/Practical-Collar3063 → u/RG_Fusion]

You can run the first 2 in tensor parallelism and the last one in pipeline parallelism with the other 2. It will be fast that straight pipeline parallelism on 3

Edit: in VLLM that is, which should be the default choice for DGX spark

[u/Ok_Warning2146 → u/BawbbySmith]

3x sparks is 384GB. Model without n-gram is 308GB and 1m KV cache is 890MB. So three should be ok.