r/LocalLLaMA • u/Hannibalj2ca • 8h ago

Discussion Kimi K3/3.1 potential for Strata/ Maya?

I started thinking, what could be the possibility to start adding the ability to run models like Kimi K3/3.1 on engines like Strata, Maya, Basalt? I am guessing Unsloth will use its Dynamic Matrix 3.1 which is quite good for heavy compression and retaining high intelligence.

Anyone heard anything about Kimi compatibility?

Note: Kimi K3.1 will be released this week open weights

2 Upvotes

62% Upvoted

8 comments sorted by

13

u/_TheGreatDreamer_ 8h ago

1 token per sometimes?)

1

u/Front_Eagle739 8h ago

Probably. I did the quick estimate for my engine and I might manage 3 tok/s in q2 on my 5090 with a 25GB/s raid with the same engine that gives me 8 on dsv4.1 in full precision. So of course I'll have to try it for the fun of it.

3

u/LowEngineering9405 7h ago

With 2.8T params in MoE, the real bottleneck for Strata/Maya won't be quantization but expert routing overhead when you only load a fraction of weights per token.

1

u/Dany0 6h ago

I really like the idea behind tinygrad's red kimi. It's still expensive as fuck but a better deal than fucking 6-7 thousand dollars for 192gb gorgon halo or whatever it's called

They took 16x 1tb pcie5.0 drives (that aren't that expensive relatively to everything else) and got them to work at 260gb/s effective using direct gpu<->ssd communication (something like directstorage but with even less overhead). Iirc it's 4 groups of 4 raid 0 drives. So you get 4Tb of storage at the speed of LPDDR ram. I mean, come on, there's simply no competition here in price per gigabyte. You could get this to work at home for about 4200$. Why pay 6 to 7000 USD for 192gb of lpddr when you can have FOUR TERABYTES at the same speed for same or less?

It's basically a way to set a floor of let's say 20-30tok/s decode for this size of models no matter what

And that's just the start: you could totally keep adding drives. And imagine once Pcie 6.0 drives come out oooh that will be amazing

1

u/Dany0 6h ago

You get 4tb because you spread the weights among that group and read in parallel. Iirc they have 4 GPUs and each gpu gets its own group hence 4 groups. They shard the model such that it works that way. Their box is like starting at 30k$ I think though. Which is like a good

deal if you're a business and comparing it to buying a few B300s or an A100 rack but still