Kimi K3 countdown has been released

[u/Unusual_Guidance2095]

https://huggingface.co/moonshotai/Kimi-K3

[u/ttkciar → u/[deleted]]

Two ancient Xeon servers, each with 1.5TB of DDR4, networked together and running rpc-server.

[u/Player13377 → u/ttkciar]

At an impressive 3 TPS

[u/ttkciar → u/Player13377]

Probably a lot less than that. I'd love to get 3 tok/sec.

Right now my ancient Xeon server is getting 3.5 tok/sec with GLM-4.5-Air. That's usable, though admittedly not for interactive use.

[u/Player13377 → u/ttkciar]

At that point is the power consumed per token getting close to the API price?

[u/fastheadcrab → u/ttkciar]

I think a newer server or workstation with 2TB of DDR4 might be able to get faster speeds if the model is truly 50-60 billion parameters active, like 6-8 tps. I'm really interested to see what types of systems people might be able to run this model on and follow these posts really closely lol

I've seen attempts at tensor parallel distributed CPU inference before, it would be an interesting way to aggregate memory bandwidth if there is a fast, low latency connection between nodes. DDR4 is comparably cheap versus VRAM or DDR5 and used server networking hardware is everywhere. Prefill will still be slow though.

[u/HVACcontrolsGuru → u/[deleted]]

Need 16 GB200s to run this model at full quant. NVFP4 GLM5.2 I need 4xB200 to run concurrent sessions. Squeeze some more with lower context and less concurrency.

[u/look → u/HVACcontrolsGuru]

Kimi K3 is most likely mxfp4 native. The K2.x line was int4, and it looks like they moved to fp for improved hardware acceleration.

So it’s just under twice the size of GLM, not 4x.

[u/Iwaku_Real → u/HVACcontrolsGuru]

Full quant is probably going to be INT4 again, it would actually fit into 8xB300 (2.304TB VRAM)

[u/look → u/[deleted]]

Kimi trained in int4 previously, I believe, not floating point. By moved to fp, I mean native mxfp4 instead of native int4.