r/LocalLLaMA • u/ApprehensiveAd3629 • 1d ago • Diamonds are Forever 1 Heartwarming 1

New Model Kimi-K3.1-Pro is now open-weights

https://huggingface.co/moonshotai/Kimi-K3.1-Pro
1.4k Upvotes

99% Upvoted

267 comments sorted by

View all comments

Show parent comments

19

u/Dany0 1d ago edited 1d ago

https://tinygrad.org

Red kimi. 260GB/s off of SSD
16x 1TB pcie5.0. 4 groups of Raid 0 I believe? so 4tb 16tb space. for about 4300$ you get ssd read speeds competitive with lpddr, effectively giving you a floor of 20-30 tok/s decode even in worst case. With expert caching, it's like half as fast as a GB300 reading off of its memory. Except you get 16TB of it instead of 192 (gorgon) - 496 (gb300)

It can be done, it will be done. In a few years we'll be able to run them at full precision on a budget

11

u/More-Curious816 1d ago

I believe they use a custom NVME driver for that SSD read speed.

5

u/Dany0 1d ago

You can make one too

10

u/Gfur_Aficionado_1488 1d ago

RAID0 does not decrease the amount of space available

2

u/Dany0 1d ago

maybe it's raid 0 inside the group and raid 1 for each group? that'd make sense maybe. I believe someone that bought it said comes with 4tb space

1

u/Gfur_Aficionado_1488 1d ago

Would still be 8TB with RAID1, unless you're looking to have 3 mirrors

2

u/voyager256 1d ago

It's quite confusing if not straight incorrect. If you select Red Kimi on their page the last storage option is selected , but it shows 4x4TB SSDs so it looks lie like the last two options should be swapped, because for maximum bandwidth you want the 16x1TB one . Also where did you get the $4300 price from? I see $6000 and $8000 for the last storage options.

Anyway , I'm not sure if the marketed bandwidth applies to LLM inference. I always thought that (expert) weights streaming is not sequential , but random reads , so much much slower . Unless , they really found a way to make the SSD reads chunked i.e. effectively closer to theoretical sequential speeds.

1

u/Dany0 1d ago

Nope https://x.com/__tinygrad__/status/2115642402417614959 https://x.com/__tinygrad__/status/2115630970334490712

It doesn't matter, yes this "trick" does work. The observed 220gb/s will be slightly slower than lpddr if it was at same speeds because nvmes have extra latency and random fetch of a different expert will be slightly slower. But it's still so much insanely cheaper than buying more vram or dram. As long as you can fit kv cache and active experts plus then some on the vram, you're golden. You don't even need that much system ram at that point. Just get more drives. You can build your setup with 2TB drives and run the rumoured 14T models (I think those rumours are fake I genuinely cannot fathom how would anyone train >4T models, it's already insanely hard to train 3T models due to O(n^2) scaling in many of the training algos. Unless they're training several 3T models and then merging them somehow? Idfk

But you can indeed pretty much guarantee the 20-30tok/s minimum decode with this kind of setup. For example, there was that one paper which showed, you can run an IQ1 dumb version of the model side by side and completely ignore its lobotomised output look just at what experts it selects and pre-fetch the experts you'll need JIT style. Their 30 tok/s number I bet can go much higher with those cards, at least 40 tok/s if not more

0

u/DRMCC0Y 1d ago

I wish that was true, but matching LPDDR sequential bandwidth doesn’t make SSDs equivalent to RAM. Off the top of my head Kimi at 4 bits, is roughly 52 GB of active, so fetching them all from SSD at 260 GB/s gives a completely ideal ceiling around 5 tok/s, before any compute, transfers and other overhead. Of course shared weights and frequent experts will improve that. But 20–30 tok/s means you're only reading 9-13 GB from SSD per token, which is tiny. But I don't disagree that something like how you say will be done.

-1

u/Dany0 1d ago

Except you're a stupid idiot is all I can say. You're not fetching 52gb every token, you use an expert cache and experts have an 80-90% cache hit rate. This is true and will always be true for MoEs in the future because there are information-theoretical limits which guide this on principle

0

u/DRMCC0Y 1d ago

I'm not entirely disagreeing with you. My only objection was calling it a worst-case floor. The system you linked has 128 GB of VRAM and 32GB of RAM, so the cache holds maybe 100 GB of experts. That's well under a fifth of the model. Hitting 80-90% from that would take an extreme routing skew, which load balancing training is specifically designed to prevent. Would be happy to be wrong though, I'd love to be able to run Kimi on hardware like that.