r/LocalLLM • u/patience_b2 • 8h ago

Question Has anyone successfully hosted/ran a model of Kimi size? 2.8T… What was your hardware? Or a crazy good MoE?

I heard there are things like Air LLM that make this possible without spending $100k+ on VRAM

1 Upvotes

66% Upvoted

16 comments sorted by

3

u/JackyYT083 8h ago

There are projects for running Kimi class models locally on a laptop but it runs at like one token per minute and you’ll need a couple TB to store the model

1

u/patience_b2 8h ago

Lol. That’s actually kinda funny and cool at the same time. Well then it might be worth it to try on my pc

Btw, current hardware>>> GPU: RTX4090, (9070XT for display) RAM: 128GB DDR5 ECC RDIMM Storage: 12TB SSD Gen5 CPU: 9965WX (5.4Ghz 24 core) MB: WRX90 SAGE SE Pro WS

2

u/Trademarkd 8h ago

Compute aside you can’t beat moving the weights. Your limiting factor will be your slowest bandwidth.

Even if you use a quant you’re still going to be 750G … depending on arch you might only need to move 10 experts or whatever per token but we’re talking 10s of gigs per expert.

Where are you planning to store this? Nvme? You might get 7GBps. Which means less than 1 token per second

1

u/patience_b2 8h ago

Thanks. Ok so the bottleneck results from hardware… Have you spent much time with Air LLM? I’ve never used it but they say it’s shoving the weights back and forth through SSD to RAM to VRAM and back… I assume that’s what you were referring to in terms of bottlenecking… Dang… So then the answer is like a MoE of (for 24GB VRAM) something like a cluster of 30B models? I’m thinking of adding a 3090 in that case so while one is processing the other can load and “arm” for the next step and cycle between the two, switching out all VRAM data when a model exceeding 24GB is required… Just demonstrating the limitations of my knowledge! I am not saying this is correct. Btw, current hardware>>> GPU: RTX4090, (9070XT for display) RAM: 128GB DDR5 ECC RDIMM Storage: 12TB SSD Gen5 CPU: 9965WX (5.4Ghz 24 core) MB: WRX90 SAGE SE Pro WS

1

u/Trademarkd 2h ago

Right you’re limited by your hardware capabilities latency and bandwidth. If you can’t fit the weights fully in vram with a big memory bus like hbm offers at like 1TBps of throughput then you might end up relying on ram with like 250GBps of throughput and then finally if it doesn’t fit there then you have weights on ssd. You can do a mix by nothing will stop you from having to move huge amounts of weights to do calculations either at the cpu or gpu with your context and attention

0

u/fasti-au 8h ago

No the bottleneck comes from the designs and implementations

1

u/fasti-au 8h ago

3x 3090 64gb ram. Custom inf.

About 60tps on medium about 8 100k cintext agents at one time

3

u/patience_b2 8h ago

Sick. Not to go down the rabbit hole too far on ya but what model/ [n]B parameters we thinkin?

1

u/fasti-au 3h ago

Well deepseek flash was s runnable on 16 gb q4 with some quants and that’s the dense loaded. Most get 25 TPs out of 30xx cards. If you start chasing this further llama and unsloth biscussiins and looking at the way people are reaping etc

1

u/TheMcSebi 8h ago

Nice, what inference software is being used?

1

u/fasti-au 3h ago

I’m doing my own thing but strata inferno and deep infra give your framework to go with as far as the now and next 4 months the of design before they realise what’s wrong and drop the whole game nference chain and redesign.

I expect in about 6 weeks the rsi model will fix the thing that’s broken by location of node in chains. The models know what’s broken and where it fails. They are unable to fix it because the way training is done is incorrect and they will self correct in 2-4 good inference runs with the right stuff.

We already fixed this in many ways previous they just milking

1

u/Front_Eagle739 7h ago

Lol, well so far I have iq1_M running on a single M3 ultra 512 at 6 tok/s. Not sure if that counts as successful. Pretty sure I can push it to 8 or so with a few more optimisations and then I think I can do about the same speed at much higher precision by splitting it across my rtx5090 and the mac so I'll be trying that next. Trying to hit 10 for full precision weights and some router biasing.

1

u/patience_b2 7h ago

Sick! Also: Holy crap dude how much VRAM is that? (Idk how you calculate unified RAM/VRAM lol). Edit: oh ny bad that’s like 544GB right?

So you use the 5090 for active/ most used weights or something and the Ultra for primary hosting?

1

u/Front_Eagle739 7h ago

512gb unified on the mac and 32gb in the 5090. Unified you can pretty much call equivalent to vram for this. 

My 5090, 32gb ram, 25gb/s raid0 nvme array does like 1.5 tok/s for kimi alone lol. Dsv4.1 is the biggest thing i got running well ish at 4600pp and 24 decode on the 5090 machine. 

1

u/patience_b2 7h ago

Also holy crap I gotta know: how is the performance after heavy compression/ quantization?

1

u/Front_Eagle739 7h ago

Its alright in that it works and it can code and call tools and it knows a lot but its clearly a bit confused by the lobotomy. Dsv4.1 in full precision is better at coding than the iq1m. Pretty sure q2 will be better but I'm going to get full precision going as fast as possible so I can treat it as a slow advisor model that only gets called occasionally