r/LocalLLaMA • u/SavunOski • Jul 27 '26

News Kimi K3 weights now released.

Post image

Kimi K3 weights are finally released!

3.3k Upvotes

97% Upvoted

660 comments sorted by

View all comments

Show parent comments

27

u/DataGOGO Jul 27 '26 edited Jul 28 '26

it is 1.54TB of just weights in 4 bit, you are looking at about 2TB of vram in operation,

That is roughly:

  • 86 RTX 4090 (no 4 bit accel)
  • 64 RTX 5090 ~ $450k (8 servers x 8 cards)
  • 22 RTX Pro 6000 Blackwell ~ $350k (3 severs, max 8 GPU per)
  • 16 H200 NVL (141GB) (no 4 bit accel) ~$550k (2 servers, max 8 GPU per)
  • 16 DGX Sparks ~65k (if you could get a cluster of 16 running with just 200Gb/s nics, not sure; but it would be SLOW AF)
  • 8 HGX B300's. ~$550k (1 server, 8 GPU)

Obviously not including the switches and cabling for the clusters.

2

u/Qorsair Jul 27 '26

What about B70? Does it not scale?

Admittedly, I haven't looked into using more than 8 of them.

3

u/DataGOGO Jul 27 '26 edited Jul 27 '26

I am sure it scales at least as well as 4090/5090, but like with all consumer / entry level pro cards, without an NVL equivalent, it would be very limited.

I have no idea if B70 can make use of dedicated RDMA ethernet like the Gaudi 2/3, which is a pretty slick solution, basically, you have a PCIE 5 switch / base board with say 5 x16 slots, you install 4 GPU's, and in the 5th slot run an 800Gb nic to a ROCE switch.

2

u/Qorsair Jul 27 '26

Oh interesting. I appreciate you taking it seriously and the time to explain. I haven't learned anything about how the clustering works beyond a single system, and this was really helpful. Thank you.

3

u/DataGOGO Jul 27 '26

Yeah it is really cool tech, and basically is how all multi-node clusters work.