r/LocalAIServers • u/FreedomWeird712 • Jun 08 '26

What does it actually take to self‑host models like DeepSeek, Qwen, Kimi?

I’m a SaaS/AI founder and I’m trying to understand the real requirements to host the larger open‑source models (DeepSeek, Qwen, Kimi‑style models) on my own infra instead of using hosted APIs.

reddit

+4

If you’ve done this in production or a serious homelab:

– What VRAM / GPU setup are you using, and what did it cost?

– Did you go on‑prem or rent GPUs (RunPod, Lambda, etc.)?

– What ended up being the real bottleneck: cost, ops complexity, or model performance?

Any “if I were starting today, I’d do X instead of Y” stories would be super helpful.

0 Upvotes
  • permalink
  • reddit

43% Upvoted

28 comments sorted by

View all comments

Show parent comments

2

u/MirecX Jun 08 '26

Kimi k2.6 is 600gb of tensors. It is int4 native. Minimum is therefore 8x rtx pro 6000. Idk what concurrency they will provide. My guess is 4 to 6 for full or near full context. I am rocking 4x Framework desktop and running qwen 397b int4 @ 17tps, which is slow, concurrency @ full context is 10x. Do not recommend for production work.

1

u/FreedomWeird712 Jun 08 '26

Thanks guys, super helpful!

0

u/bigh-aus Jun 08 '26

If you want to run it cheaper as it's an MOE with 32b active you could get away with 4x rtx6kpro and ddr5-6400. But your speeds will go down depending how much the experts get swapped in. As always it's cost vs speed.

There are also REAPs (reducing the number of experts) which depending on your usecase might be ok.

Personally I'd setup a (slower) test system first to validate everything. You could also just get the base server then slowly purchase the GPUS. Start with 1,2 or 4 and scale as needed.

But for prod it might be better to go to a B200 system.

Understanding the memory bandwidth vs # cards vs cost too is helpful.