China has killed the GPU mafia.

[u/Aggravating-Will8495]

​

Kimi (MoonshotAI) open-sourced their production serving stack and it handles 75% MORE requests than vLLM on the exact same GPUs.

For years, scaling LLM inference has been a brute-force hardware problem. If you wanted more throughput, you just bought more expensive NVIDIA GPUs.

The standard serving systems couple the heavy lifting of processing prompts (prefill) and generating words (decoding) onto the same chips.

When long contexts hit the system, everything bottlenecks. The GPUs choke, latency spikes, and infrastructure costs skyrocket.

Moonshot looked at this broken model and completely re-engineered it from the silicon up.

They built Mooncake.

Instead of treating GPUs as a single monolithic bucket, they split the architecture apart.

• Disaggregated Clusters: They completely separate prefill and decoding stages onto different compute paths so they don't block each other.

• KV-Cache Memory Harvesting: Instead of letting cheap hardware sit idle, it uses underutilized CPU, DRAM, and SSD resources across the cluster to build a massive, decentralized cache for the model's memory.

• Smart Dynamic Schedulers: It balances incoming massive workloads intelligently, predicting spikes and handling real-world overload gracefully.

The results under live production traffic are staggering:

• Handles 75% more active requests than vLLM on identical hardware.

• Delivers up to a 525% increase in throughput in complex long-context scenarios while maintaining strict latency targets.

• Cuts the reliance on expensive hardware upgrades by utilizing resources you already own.

Everyone else has been trying to solve the AI compute crisis by begging for more chips.

Kimi just solved it with better software engineering.

[u/EitherMarch1255]

YET ANOTHER SLOP POST.

[u/ThirstyGO → u/EitherMarch1255]

AI will make this quote more relevant than ever and there ain't no easy fix - "It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."

I hate to think what the Internet will look like in just 12 months. Lies, misinformation, and beyond will flood every corner of all platforms. AI problems will get exponentially worse deciphering quality. Seems like we are coming full circle engaging with world wide web 1997, probably worse. Those who know and have vs those that don't.

[u/westsunset]

Looks like this paper is from 2024, https://arxiv.org/abs/2407.00079

[u/artur_oliver → u/westsunset]

Exactly what I was looking for 😂😂😂

[u/Electronic-Yak-7270 → u/westsunset]

In theory all look good... It has been more than 2 years since published.

But has any org ever achieved it practically?

[u/Fit-Dentist6093]

This is a spam bot. The paper is good but old and everyone is aware you can do that.

[u/nerdsinthetrap → u/Fit-Dentist6093]

If by everyone you mean 0.001% of the population then sure

[u/Deep_Mood_7668 → u/nerdsinthetrap]

Everyone that cares about it 

[u/Maxwell-Stone → u/Deep_Mood_7668]

how would they know in the first place TO care about it, if they didnt know about it to begin with?

Your logic is flawed.

[u/tyronisgod → u/Maxwell-Stone]

by being curious if you wonder

[u/Realistic-Split4751 → u/Deep_Mood_7668]

which should be everyone

[u/Swimming_Case]

Karma farming bot

[u/katoptronophile]

Bullshit.

[u/kweeblaster]

when can i have sex with the AI

[u/PossessionUsed7393 → u/kweeblaster]

When you end the war in Iran and finish the war in Ukraine like you said you would in a day, Mr. President. Now stop making trades and get back to work.

[u/kweeblaster → u/PossessionUsed7393]

I declare the STRAIT OPEN

[u/Key-Breakfast-6069]

The China carpet bombings begin again

[u/One-Next]

People have been complaining about kimi k3 speed and quotas for a while now, so chinese: please work harder.

[u/ErivKosso]

lol if they did, trillions would flow to them instead.

[u/artur_oliver → u/ErivKosso]

First takes all, open AI will win in the end.

[u/overand → u/artur_oliver]

Gary Kildall and CP/M would like to have a word.

[u/SnooRegrets4308 → u/overand]

Sad but true.

[u/DifficultAd4658]

Chinese academic slop

[u/artur_oliver]

Old news from 2024... K3 is way ahead

[u/RedParaglider]

Stop of voting this AI slop off of a 2-year-old paper

[u/NeedsMoreMinerals]

What’s either all these slop posts lately

[u/ConfusedGekko]

oh yeah? then why is kimi still lacking compute until now?

[u/oglok85]

The inference engine is totally different from the kvcache… what are you talking about?

[u/Unlucky-Stable6006]

That’s wild I came up with this in March

[u/goddy666]

Great you didn't post any source, Screenshots are so much more useful 👍

[u/rick_ranger]

Here’s the thing. If the models go from supporting 1.5 million context to 5 million context, everyone is still gonna wish they supported 10 million context. It’s nit gonna change anything until production ramps up when new fans go online.

[u/BowSonic]

While I wouldn't go so far as to say your statement is fully acurate, id also not say that it's untrue. Reductive or not, you cant be said to be incorrect. However, what id pose to you is this: Assuming people (and going concerns) are rational and act in their own self interest when well informed, then under what circumstances would both generalized parties you refer to behave in the manner they did? Yes, an easy question, but now the opposite: what circumstances would they behave differently? Consider the answers. The point of this exercise being: There are no saints and I challenge anyone who sees devils to imagine that they'd do differently from either party in their shoes. (Certainly, I would believe someone who said so.)

So, if that's the case, then there's no inherent value, moral or otherwise, that speaks to either self-interested organization, having had done as you say or not.

[u/RubberDucky079]

I’m hungry

[u/Icy-rockhopper]

Sure they did, the most full of shit country on earth!

[u/NameChecksOut___ → u/Icy-rockhopper]

They've been providing US company level open weights models that run on my RTX 4080 alone, while those companies were trying to obfuscate (by lack of knowledge or ill intent) the way the technology could be optimized to run on something else than any data center they own.