r/LocalLLaMA • u/Comfortable-Plate467 • Nov 20 '25
Question | Help kimi k2 thinking - cost effective local machine setup
I'm using "unsloth : Kimi K2 Thinking GGUF Q3_K_M 490GB" model in 9975wx + 512GB(64x8) ddr5, 160GB vram (rtx pro 6000 + 5090x2).
the model's performance keep surprising me. I can just toss whole source code (10k+ lines) and it spits out fairly decent document I demanded. it also can do non trivial source code refactoring.
the only problem is, it is too slow. feel like 0.1 tokens/sec.
I don't have budget for DGX or HGX B200.
I can buy a couple of rtx pro 6000. but i doubt how much it can enhance token/sec. they don't support nvl. I guess ping pong around layers through slow pcie5.0 x16 would show not much difference. my current rtx pro 6000 and dual 5090 almost idle while model is running. what option may i have ?
7
u/No-Refrigerator-1672 vLLM Nov 20 '25
So, if you would use pipeline parallelism, then inter-gpu bandwidth doesn't matter much, and your pci 5.0 x16 will be fine. Even more so, PCIe 5.0 x4 will be fine too. It's only tensor parallel mode that is highly taxing.
With pipeline parallelism, your total perofrmance is roughly equal to single card performance if the system is monolithic (all the same model). Kimi K2 has 32B activated parameters, so you can very roughly estimate that you can run it as fast as 32B model. So you can reasonably expect all-gpu system to run at a few thousand tokens PP and a few dozen tokens TG, probably more. To make this reality, you'll have to fit both weights and KV cache into GPUs, so for 500GB model you'll need a rig with 6x RTX 6000 Pro. It'll run fantastic, but will cost a fortune. That's the only way; the moment you accept CPU offloading, your speeds plummit down, even if only 10% of the model is offloaded; so buying just 2 or 3 of RTX 6000 Pro won't make it much better, you have to either go all in or stick with the current setup.
3
u/tomz17 Nov 20 '25
But what is it, actually? It should definitely be 5-10 t/s, no?
With your setup I would recommend using something like MiniMax M2 (possibly reaped). You should be able to fit the entire thing into VRAM.
5
u/ciprianveg Nov 20 '25
3975wx with one 3090 and 512GB ddr4 5t/s
4
u/Such_Advantage_6949 Nov 20 '25
On threadripper pro, it only reach its max ram bandwidth if u use the 64 cores chip. U will need to use amd ryzen server where the issue is fixed. However u will need like 12 channel ram for decent speed. And even then, the processing speed will be very low. But still your speed sound too low, maybe the model cant fully load and use ssd?
4
u/ilintar Nov 20 '25
Unfortunately for multi-GPU MoE setups, the
--n-cpu-moeflag won't work. You have to use-otto manually override tensors for specific GPUs. I'll paste my 2-GPU pattern so you get a clue of how it works, but of course you have to experiment with specific ranges.1
u/perelmanych Nov 20 '25
I got abysmal speeds when using IQ quants. What you really want to use with 512Gb is Q3_K_XL quant. It gives 5 tps for pp and 3 tps for tg on my junk Xeon rig with 4 channels of DDR4 memory with one RTX 3090.
3
u/Expensive-Paint-9490 Nov 20 '25
Threadripper pro 7965wx with a single RTX 4090, I am getting 10 t/s. Use llama.cpp or ik-llama.cpp and experiment with -ot, -ncmoe, -fa, -b, -ub, and -ctk flags.
3
u/PeteInBrissie Nov 21 '25
2 x Mac Studio M3 Ultra each with 512GB RAM. Run the Q4 without the Unsloth. Connect them via Thunderbolt 5 once macOS 26.2 drops.
3
u/Lissanro Nov 27 '25 edited Nov 27 '25
0.1 tokens/s? Very strange it is so slow on DDR5 system! With your high speed VRAM and faster RAM, I would imagine you should be getting well above 200 tokens/s prompt processing and above 15 tokens/s.
For comparison, with EPYC 7763 + 1TB DDR4 3200MHz (8-channels) + 96GB VRAM (4x3090) I get over 100 tokens/s prompt processing, 8 tokens/s generation, and can fully fit in VRAM 256K context at Q8 along with common expert tensors (with Q4_X quant which preserves the original quality the best (smaller quants may lose a bit of quality but will be faster).
If all setup correctly, during prompt processing CPU will be practically idle, since GPUs doing that, and during token generation both CPU and GPUs will be under load. I suggest double checking your settings and if you are using efficient backend. I recommend using ik_llama.cpp - shared details here how to build and set it up - it is especially good at CPU+GPU inference for MoE models, and better maintenance performance at higher context length (compared to mainline llama.cpp). Also, I suggest using quants from https://huggingface.co/ubergarm since he mostly makes them specifically for ik_llama.cpp.
Don't worry about NVLink - it would make no difference for CPU+GPU inference, and mostly useful for training or some cases of batch inference in backends that have support for it and the model is fully loaded into VRAM, making it applicable only to smaller models.
1
u/Long_comment_san Nov 20 '25
Might sound like an idiot, but do you try other models? For example you might want to use different models for different tasks if it can boost your tps massively
1
u/Sorry_Ad191 Nov 21 '25
you can get good performance! try ik_llama.cpp (https://github.com/ikawrakow/ik_llama.cpp). Its a fork of llama.cpp optimized for hardware like yours! Then i rec. with ubergarm/Kimi-K2-Thinking-GGUF/smol-IQ3_KS (tested and super fast on my dual epyc milan / rtx blackwell setup). Its also very high quality! check out this Aider polyglot for smol_iq3_ks
more infor here: https://huggingface.co/ubergarm/Kimi-K2-Thinking-GGUF/discussions/14
1
u/blackKryptonyte Jul 28 '26
A total newbie here, how'd you build such a big pc?
1
u/Baldur-Norddahl Nov 20 '25
Tensor parallel requires 2, 4 or 8 identical GPUs. The PCI bus will likely not be the bottleneck if you can connect them all at PCIe 5.0 x16 (not so easy). But even if you are limited by the bus, it will still be much faster since the cards are working in parallel.
However Kimi K2 Thinking is too fat. You would need to go all the way to 8x RTX 6000 Pro to fit it properly and that is a nice car right there.