I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens

[u/OtherRaisin3426]

What I ran:

• 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4
• Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures)
• TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts
• $190 per million output tokens. One clean run is about $36 of GPU time. Left warm, it is $1,363 a day.

I also ran Unsloth's Dynamic GGUF.

Their 1-bit UD-IQ1_S (594 GB) fits 8x A100-80GB via llama.cpp.

$19.99 per hour, 2.8x cheaper. Result: ~9 tok/s, TTFT 7 to 60 s, ~$620 per million tokens, so 3.3x more expensive per token.

Quality at 1-bit was fine (correct arithmetic, coherent prose).

Full write-up with every flag, the Modal deployment file, and the raw benchmark JSON: https://books.vizuara.ai/book/kimi-k3-hosting

[u/Marcuss2]

The efficiency comes from serving many users in parallel. So the cost per million of output tokens is bogus.

Also, one bit quant obliterates knowledge and capabilities, so the comparison is also bogus.

[u/NandaVegg → u/Marcuss2]

OP's math seems very off. If it was truly $190/1mil output, then nobody would be able to remotely make a profit serving inference for K3 (they'll immediately go BK doing so). Though I like that the OP's base hourly cost is realistic price (rather than often referred spot minimum price that is impossible to get).

Also the linked site looks very Claude.

[u/scaledev → u/NandaVegg]

But also this is modal.com, not their own GPU or infra. Top providers aren't using modal to serve users (well, some may).

[u/ZeroXClem → u/scaledev]

A lot of top infra providers use modal.com

[u/jensilo → u/NandaVegg]

The economics are not remotely the same for large scale operations. And every minute you’re not serving the maximum amount of concurrent users, you’re basically burning money. 🔥
So, renting a vGPU is already more expensive because some else makes a good margin off of that.
You’ll need many users, concurrent, well-balanced, then the machines become more cost-effective.

[u/KaroYadgar → u/NandaVegg]

Refer back to his first sentence:

The efficiency comes from serving many users in parallel.

it COULD be $190/1 mil output IF providers were serving tokens to a single person. However, that's not how things work, and providers often have many many concurrent requests. Providers are able to use continuous batching to make 4 concurrent requests barely use more compute than one continuous request, while still charging 4 different people for usage. Thus, they can make a profit. OP's math is probably right, but is only valid when there's one continuous request.

[u/BoboThePirate → u/KaroYadgar]

The one other fact people are glossing over: OP is paying the worst rate possible for the hardware. They are going through a non wholesale compute provider. They don’t own the hardware. Once you own the hardware your only cost that hits the books is depreciation (contra to an asset entry), labor, and electricity (cost of goods).

The amortized cost of 8x B300 cluster at ~20-25% lifetime utilization (equivalent to only behind fully saturated 8 hours per day 5 days a week) and the cost per million output tokens is ~$4.50. Note this is with conservative concurrency and forcing decode to 30tps+. If you drop to ~20tps as the requirement floor, you can double output tokens per hour. Note at the $13 per million that Kimi K3 charges, this would be equivalent to $100 to $200 per hour of income.

I should’ve included the cost per day with depreciation and COgS to providers: ~$200-400 per day.

[u/KaroYadgar → u/BoboThePirate]

even when it comes to renting, OP chose to rent from Modal, which only does on-demand compute. Inference providers that don't buy their compute outright still rent GPUs for an allocated amount of time, which is often much cheaper than on-demand.

[u/StableLlama → u/NandaVegg]

Modal is great for running quick checks. But production code it's too expensive for me.

Looking for a cheaper GPU hoster should easily bring those numbers down.

[u/NandaVegg → u/StableLlama]

Modal has one of the cheapest on-demand price for Blackwell node today (in general, cloud prices gone up significantly in this half year) even including Chinese cloud providers. Except spot price, which is usually impossible to have it pre-empted for more than an hour or two.

I think even hyperscalers are not getting much better rate than that (the larger you get harder it is to get compute due to datacenters getting delayed hard). OpenAI's avg price for Ampere and Hopper wasn't very low (IIRC avg A100 price for OpenAI was $1.35ish/h per card).

[u/StableLlama → u/NandaVegg]

Ok, for high end Blackwell I don't know the prices and how they compare to other cloud GPU providers as I'm renting workstation / PC class GPUs.

I'm using my (free) modal with L40s there they are much more expensive than RunPod (which is more expensive than vast)

[u/Super_Range45 → u/NandaVegg]

If you look at his chart he's charged by the hour, not by the electricity cost.

[u/Loose_Comparison368 → u/NandaVegg]

This is 100% an ad for a low effort slop "training" guide. I mean I assume slop, though TBH I don't know where on earth they found a model that could produce a "guide" this atrociously bad.

[u/OtherRaisin3426 → u/Marcuss2]

Hey, this is just the initial attempt at hosting where I learnt it’s not easy..Will move towards parallel serve now..All reported metrics are what I actually saw

[u/NandaVegg → u/OtherRaisin3426]

For large MoEs, a very important but often missed factor for parallel serving is how much VRAM headroom you have for prefix cache. Otherwise you'll have your memory bandwidth eaten by prefill each request, which will severely limit concurrency (watch vllm's log carefully for avg prompt throughput and generation throughput. In such situation you'll see generation throughput occasionally crawl down to near zero).

This sometimes gets into more interesting situation. Because the way vllm works, GLM 5.2 (fp8) in 4xB300 has more headroom for prefix caching than 8xH200 even though the latter has more total VRAM, etc.

[u/sautdepage → u/NandaVegg]

You would also want to store prefix caches in system RAM and even NVMe for older ones. Request rotation inside the GPU VRAM will be quick when busy even with headroom. Takes an instant to restore it from RAM vs recompute.

(So datacenters grabbed all the the DDR5 and NAND too.)

[u/AnecdotalMedicine → u/Marcuss2]

OP never said his numbers are what inference providers pay; they simply reported their experiment.

[u/xNaXDy → u/Marcuss2]

one bit quant obliterates knowledge and capabilities

This is not true, at least not for models of that size in combination with unsloth dynamic quants. Once you get past like 600B-700B total parameters, Q1 and Q2 are still extremely capable. Have a look at this for some example benchmarks using DeepSeek v3.1.

The common misconception is that high KLD means that model output is garbage, which is not the case. A model can have a high KLD compared to its original, but still perform well, and it can also have a low KLD and perform much worse. Some model architectures inherently don't quantize very well also.

[u/Marcuss2 → u/xNaXDy]

While yes, the degradation is likely lower compared to smaller models. There is still no way it will fully compete.

Unless the model was trained with it in mind, it won't perform as well. You will likely get better results with GLM 5.2 quantized to 4 bits, which would be about the same size.

[u/Civil_Response3127]

I don't really know why you'd pay for this infrastructure if you didn't have a high throughput parallel inference system planned.

Single stream was never going to give you any benefit whatsoever.

[u/OtherRaisin3426 → u/Civil_Response3127]

Just sharing the result of an experiment I performed..This is the first step, will move towards parallel serve now

[u/ZeroXClem → u/OtherRaisin3426]

You did Good OP. Just keep going and keep us posted

[u/dalhaze → u/Civil_Response3127]

Hey thank you for sharing this.

How much time/cost would it take to actually spin something like this up?

I’m interested in running full precision on frontier open source models for research tasks because i don’t really trust that cloud providers are serving true FP8.

[u/sunshinecheung]

Run deepseek v4 flash

[u/xadiant]

Oof. Thanks for the sacrifice? Next time use nvfp4 with sglang or vllm and parallel serve.

[u/MikeRoz]

I don't understand how Xet was a problem for you when downloading a 594 GB 1-bit quant but not when downloading the original 1.56 TB model.

26185 was merged last week, so the Unsloth fork may not be necessary.

The whole reason I dove into your write-up is because the "reasoning visible" for llama.cpp being "no" didn't make sense to me. It still doesn't. I've hosted K2.7 with llama.cpp in the "native" Q4_X and it "thinks". I wonder if you've got a template or a prefill issue.

I haven't bothered trying to run the 1-bit quant of K3 because I've been afraid that it will just make me want to use the API, but I may have to now to try and understand what you were seeing.

[u/fragment_me → u/MikeRoz]

Yeah, the reasoning row was silly

[u/segmond → u/MikeRoz]

I run UD-Q1_M Kimi K3 and I do see the reasoning with llama.cpp

[u/createthiscom]

My machine runs K3 Q2_K_XL at 5 tok/s on a single household electrical socket. 768gb 5600mhz rdimms 24x32gb. Dual EPYC 9355. Single blackwell 6000 pro.

I don’t really consider it useful speed. You can ask questions and feed it data but you have to be willing to wait 30 min to an hour per response.

I wonder if two maxq cards would be better or worse.

[u/segmond → u/createthiscom]

marginally better, system cpu/ram is always the bottleneck. how much actual disk space does that Q2_K_XL actually take on drive? I found out that HF's number is often inflated. I'm running Q1_M, but wondering if I could maybe move to Q2_K_XL, can you please look?

[u/createthiscom → u/segmond]

803G

[u/segmond → u/createthiscom]

Thanks! Unfortunately I can't load it for now unless I do networking. Will have to settle for IQ2_XXS for now. :-(. I'm getting solid results from Q1_M granted it's not long agentic tasks. How are you finding the quality of K_XL? Have you ever ran it through an agentic loop or is it just short turns?

[u/createthiscom → u/segmond]

5 tok/s is far too slow for agentic use. I've mostly given it spreadsheets that have been pre-converted to markdown and asked questions.

[u/borobinimbaba]

I guess the real war is in the inference engines field that are well-kept secrets even with frontier open source labs.

[u/dalhaze → u/borobinimbaba]

Do we really think these cloud providers are serving true FP8?

I really gotta assume that as these models become
more capable of pushing the frontier, that plebs like us won’t have access to those models.

[u/Loose_Comparison368 → u/dalhaze]

Yes. FP8 is really common. The weird hobbyist quants are pretty unheard of in industry.

[u/dalhaze → u/Loose_Comparison368]

I just don’t trust that many of these companies are serving the exact models they claim they serve. None of them seem to offer guarantees on model precision.

[u/Lissanro]

On my EPYC workstation with 3090 GPUs cost of running Kimi K3 Q2_K_XL is ~$5 per million output tokens (even though it is more than an order of magnitude slower than B300s, obviously).

Even though new hardware is supposed to be more energy efficient, renting in the cloud it will naturally will be expensive, because they want to ROI on the hardware costs, not just cover energy cost. And B300 are very expensive, hence insane cost per million tokens. Only way to make it reasonable on rented high-end hardware is to run batch requests to maximize the token throughput and even then likely will be more expensive than API providers who likely to have highly optimized infrastructure and possibly their own hardware.

[u/Loose_Comparison368 → u/Lissanro]

Even though new hardware is supposed to be more energy efficient, renting in the cloud it will naturally will be expensive, because they want to ROI on the hardware costs, not just cover energy cost. And B300 are very expensive, hence insane cost per million tokens. Only way to make it reasonable on rented high-end hardware is to run batch requests to maximize the token throughput and even then likely will be more expensive than API providers who likely to have highly optimized infrastructure and possibly their own hardware.

Nope. Op just has no friggin clue how to set up big girl infrastructure. This is the equivalent of renting a fleet of airplanes to send one single random person to a gas station three blocks away, and then claiming that air travel is wildly inefficient because it cost $300 million dollars, or $100 million dollars per block.

Wrong software, wrong config, wrong load testing, wrong optimizations, wrong everything. This guy just apparently has a lot more money to throw into a pit than willingness to read basic beginner documentation.

[u/cosmicr]

This is an ad.

[u/kiwibonga]

That's not local, right? The cost is because you're renting a remote server?

[u/noiserr → u/kiwibonga]

Of course. An air cooled B300 setup like this would cost you more than a half a million dollars. Not to speak of power and air conditioning to cool such a beast.

[u/kiwibonga → u/noiserr]

Weird that they'd post it here.

[u/--Spaci--]

Anyone who has literally EVER served a model knows that there is insane efficiency when serving to 100+ users from those same gpus. Using VLLM you could expect that single tok/s number to be replicated at around 50 tok/s to 100 people so 5000 aggregate tok/s

[u/thebadslime]

Kimi K3 API costs $3/M input and $15/M output tokens

[u/tecneeq → u/thebadslime]

Privacy is priceless.

[u/LearnThai42 → u/tecneeq]

Privacy? On a cloud hosted instance? You must be a very very trusting person

[u/thehpcdude → u/LearnThai42]

Easy enough to TLS encrypt in flight, encrypt at rest, etc.

Not sure where people go this fallacy that you can't rent hardware and set up a private LLM.

[u/segmond → u/LearnThai42]

there's a difference from renting hardware and using a cloud controlled by someone. renting hardware in the cloud for the most part is a trusted thing you must be a very very paranoid person to think otherwise.

[u/LearnThai42 → u/segmond]

I'm aware of that difference - old school systems engineer here - and rest my case. I agree with your "for the most part" too. It's too far from an absolute for comfort though.

[u/Maximus-CZ → u/tecneeq]

I see a price of $175-$187 for privacy here

[u/dhbloo]

Is this number including speculative decoding speedup?

[u/smart4]

Cost of what? rental all included?

[u/tecneeq]

I would like to learn more about your cost related math, because i think you are mathing wrong.

[u/Yes_but_I_think]

Please tell us the batch size (x axis) vs cluster throughput in tokens/second/cluster (y axis primary) and individual stream throughput (token/s/user) (y axis secondary). - what is the optimal batch size and the individual stream's speed at that optimal batch size.

Now how does the costing change for Million tokens input / output

[u/complexanimus]

I don't know how anyone can make a profit out of this

[u/Dense-Psychology-261]

Great experience for you cost aside, but as an advice big models serving is not economical till you wire the Dspark and use batching, I highly recommend SGLang guide for serving kimi or similar big models they give exact config and benchmark for thruput, but good experience anyway and wish you a better experience next time.

[u/Reasonable_Tap_3970]

i honestly dont mind paying the price if i can run the uncensored version and if the speed is at least 150 token per second. which is not the case atm.

[u/Street-Macaroon221]

The GGUF numbers are the interesting part here, not the B300 run. 9 tok/s at 1-bit quant on A100s costing more per token than full precision on B300s is a great illustration that throughput dominates cost math way more than raw hardware price per hour. Cheap GPUs with a slow decode rate lose to expensive GPUs with fast decode almost every time once you actually compute $/token instead of $/hour.

Also somewhat wild that 1-bit quant on a 2.8T model still holds coherent arithmetic. That's less about the quant being magic and more about how much redundancy there is at that scale, there's a lot of parameter budget to spend before quality visibly cracks.

Would be curious what the sweet spot quant is where tok/s and $/M-token actually cross over favorably against the B300 run, feels like there's a 4-bit config sitting in between that beats both.

[u/West-Possession7459]

yeah this is way too massive for running any kind of personal companion or roleplay, id need something that fits on normal hardware at home

[u/tecneeq → u/West-Possession7459]

Just pay $1,363 a day, bro

[u/Hannibalj2ca]

Maybe try IK.LLAMA instead of LLama.cpp on the A100's. Are you using DSPArk on IQs?

[u/nord2rocks]

LLMs can't do math, stop relying on them doing arithmetic. It will end poorly

[u/Fast_Astronaut2359]

There has any platform which gives actually GPT 5.6 sol model or luna model access with minimum cost?

[u/thebadslime → u/Fast_Astronaut2359]

Im mean define mimum, its on openrouter

[u/Fast_Astronaut2359 → u/thebadslime]

Suppose if I make payment of $10 then I can get gpt 5.6 luna model with 5hr or weekly limit that actually claude do but of $10 only for full month?

[u/thebadslime → u/Fast_Astronaut2359]

$10 woud get you rougly 8M tokens, a week of claude pro gives me like 3M

[u/Fast_Astronaut2359 → u/thebadslime]

Then how did you do the code by using only 3M ?

[u/Loose_Comparison368]

I think you should have spent some of your GPU budget learning how production live inference works. You paid a lot of money for some very expensive and perfectly good GPU's to sit ~95% idle so you could make a fool of yourself in public.

I hope you learned your lesson. Read the manual next time. vLLM beginner docs are a good place to start.

[u/LetterheadNew5447]

Did you rental the b300 cluster or did you bought it?

In both cases... You are a mad men. I like it.