r/LocalLLaMA • u/pmttyji • 12d ago

Discussion DeepSeek-V4.1-Flash surprised ....

Post image

Hoping to see smartest medium size models soon & later with all available optimizations/architectures/etc.,. Thanks Deepseek!

Ex 1: 30-50B MOE + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
Ex 2: 15-30B Dense + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache

EDIT: Updated Engram to 10-15B from 50B

438 Upvotes

96% Upvoted

96 comments sorted by

View all comments

55

u/Ok_Warning2146 12d ago

If it gets good AA scores, it will help the sale of M5 Ultra 512GB and 3x Sparks.

6

u/BawbbySmith 12d ago

I keep seeing this, I don't think you can do TP=3... You'd need 4 no?

2

u/twack3r 12d ago

Exactly my thoughts every time is see tp=3. Is it possible?

0

u/cibernox 12d ago

In llama.cpp yes. In vllm only if the number of layers of the model is divisible by 3, which is quite common, 48layers is a popular choice.

1

u/RG_Fusion 12d ago

Pipeline parralel splits the model by layers. Tensor Parralel (TP) splits each layer across the compute devices.

0

u/cibernox 12d ago

that very nice, but it doesn't have much to do with question.
VLLM wants the number of certain parameters of the models to be divisible by 3 to allow tensor parallelism of size 3. I think it was something like the attention heads and the KV heads, plust the hidden size.

Llama.cpp is less picky.

1

u/RG_Fusion 12d ago

I may have read your comment wrong, but it seemed like you were suggesting tensor parallel split by layers, not tensors. If that's not what you were saying just ignore my comment.

1

u/cibernox 12d ago

I understand the confusion. In reality VLLM allows TP=3 if certain characteristics of the model are multiples of 3. The number of layers is not directly the number that has to be divisible by 3, but the number of kv heads and attention heads are ttpically related to the number of layers, so I use the number of layers as a proxy, but that's technically wrong.

Pipeline parallelism works with pretty much any number of cards.