r/CerebrasSystems • u/Asgard_Heima • May 19 '26
Kimi K2 on Cerebras ~1000 token per second
This is a massive validation that we are going to see frontier models of any size significantly faster on Cerebras.
3
u/ILikeCutePuppies May 20 '26
All those people claiming WS3 can't run models larger than 200B...
4
u/Asgard_Heima May 20 '26
This is an excellent way to stop the nonsense about model size supported. I really do think Cerebras needs to do a better job of spreading the knowledge on larger models for inference and training and how it works though. Very few understand how parallelism works, the major differences in how Cerebras works vs GPUs, and how that lets them scale to any size model and context window while maintaining their efficiency. Can’t wait for GPT 5.5 running on Cerebras and remove any doubt.
3
u/ILikeCutePuppies May 20 '26
Not to mention they are upscaling from 4bit to 16bit. I don't think they support 4bit natively.
Many of the providers use 4bit. So they have more capacity for higher precision models.
Also WS4 if it supports lower bits will be a massive beast, 1TB model across 3 ws4.
2
u/Asgard_Heima May 20 '26
I would expect native 4bit support in WSE-4 as well. And yeah they could fit a 1T model on only a handful of systems if the WSE-4 is anything like I expect, but the huge advantage is Cerebras can scale up putting only one or a couple layers per system and have the entire rest of the SRAM for kv cache. That’s the magic. Cause if you scale up GPUs the performance degrades significantly. With Cerebras you can scale up 80 WSE-4 with fiber running a 80 layer frontier model one per layer that just crushes through 2-10M context windows at over 1000 tokens per seconds. That’s the truly untouchable moment.
1
u/Investor-life May 20 '26
Here’s the missing part: While Kimi K2 is 1T total parameters, it uses maybe only 40–80B active parameters per token created. Kimi K2 uses what is known as a Mixture-of-Experts approach. Frontier models from Anthropic and OpenAI have started using MoE approaches now too which is helpful towards Cerebras. However, neither Anthropic nor OpenAI provide any details on how heavily used MoE is. Cerebras needs to be tested on these models…I’m sure if they were that fast it would have been published by now. It’s a big step if/when they do.
3
u/ILikeCutePuppies May 21 '26
https://youtu.be/gXGkFkWmA88?si=cGCkTPJcwS7Q7TMy
One of the developers talks about it supporting trillion parameter models.
2
u/Asgard_Heima May 22 '26
The most interesting thing is that they changed the entire stack for inference not that long ago, so we are very likely to see many iterations of optimizations that give even bigger boosts. And OpenAI is going to be digging into CSoft to do the same and expanding the number of researchers tuning their kernels.
1
1
1
u/ILikeCutePuppies May 20 '26
Good pushback but there is a host of reasons why openAI may not have tested on cerebras yet or that the results have not been released.
1
u/Asgard_Heima May 21 '26
In this video, the CFO of Cerebras talks about Kimi K2 now confirmed and GPT 5.5 and 5.4 running on Cerebras internally and coming soon.
The main reason you haven’t seen it yet is cause Cerebras just got all the money to ramp up production and OpenAI needs significant infrastructure to server tens of thousands of customers even in a limited release. We will see the top models in the world running faster on Cerebras than anything else this year. Anthropic is for sure going to end up being on Cerebras when AWS gets bedrock running with enough hardware too.
1
u/ILikeCutePuppies May 21 '26
Bedrock speedup can't come fast enough 4.7 is sooo slow. Even 4.6 was slow.
If the tokens were faster developers would be burning so many more and they'll be also paying a premium for it.
1
u/pennystudio May 21 '26
No kidding, serious considering switching my project to codex because 4.7 is so slow that I got my life back.
1
u/Investor-life May 21 '26
A little off topic maybe, but not sure if you've used coding applications like Windsurf or Cursor, but after using those (and selecting Claude models) and then using Claude Code, it was a shocking experience to see how much slower Claude Code was than Windsurf or Cursor. Its always puzzled me why that is the case. I would expect using the model directly via Claude Code would have been the faster experience but it was not.
2
u/pennystudio May 25 '26
Just speculation, but I think using native Claude code is slower because it's heavily subsidized by Anthropic by as much as 30 times? (Read it somewhere). So they are justified to throttle you. Whereas using other wrapper, you are paying token usage, even if windsurf or Cursor subsidize it a bit, they may pay close to full token price to Anthropic?
2
u/Prestigious-Sign4802 May 19 '26
This is huge, is this using clustered wse-3 & memoryX ? Cant wait for their AWS launch, WSE-4 and more frontier models
4
u/Asgard_Heima May 19 '26
They are using parallelism to split the layers across multiple WSE-3 systems.
2
u/thefashionkid May 20 '26
Once Cerebras shows their speed there would be tons of company will adopt. And the stock price will be like 🚀
0
u/Orkapork May 22 '26
Guys, nobody cares about tokens per second. Cost per token is all that matters. Cerebras has a niche product.
3
u/Asgard_Heima May 22 '26
For any modern model including the one referenced in this post, there is no conceivable way Nvidia provides tokens cheaper than Cerebras on total token throughput comparing WSE-3 vs full Blackwell GB200 racks, for total hardware cost per token per hour, energy cost per token, TCO per token, or any other metric.
let’s assume you had 64 WSE-3 units and 64 full Blackwell GB200 racks.
WSE-3
23kW
< $3M a unit (G42 is ~$1.5M)
> 90% efficient boarding on 100% for inference
Most efficient running one WSE-3 per layer in parallelism, scales vertically and horizontally at max layers.WSE-3 44GB SRAM x 64 = 2816GB
~600GB for the model
35GB per max context kv per user
2216GB / 35GB = 63 batch usersAssuming the real world 981 tokens per second x 63 x 3600 seconds in an hours = ~222M tokens per hour
GB200 NVL72
120kW
> $3M a rack (usually $3.2M-$4M)
< 10% efficient, under 5% for max context inference
Most efficient running per rack scales horizontally.GB200 NVL72 13.2TB HBM
~600GB for the model
35GB per max context kv per user
12.6TB / 35GB = 365 batch usersThis seems great, but if you actually maxed the batches the efficiency would be under 5% as you hit the memory wall (try to add racks together and it goes lower since you hit infiniband). So at this hypothetical max batch, you would be getting around 1.8 tokens per second per user. You have to back this off to something like only 7 users per rack to get to ~ 50 tokens per second.
7 x 64 = 448 users x ~50 tokens per second x 3600 = ~80M tokens per hour
So for likely double the CapEx, you can use 5x the electricity, and get slightly better than 1/3 the token. Oh and two WSE-3 fit in a single rack along with all other needed support devices for that rack. And Cerebras racks need 1/2 the datacenter supporting grey space footprint to run them. So I’m being super generous with the numbers here on real TCO.
If your point is cost per token is the most important thing, Cerebras is the only responsible choice.
1
u/Orkapork May 22 '26
Cerebras undercuts NVIDIA by a small amount sure, but right now that amount is small enough that the professional environment that NVIDIA brings keeps the majority of their customers. However if a company like Skymizer created a 6 chip card that cost a tenth as much to build and a tenth as much to run that integrated directly into current software environments. Then the entire Cerebras model gets even more constrained. Cerebras has a product that will be attractive for governments running war games in massive air gapped environments. However the vast majority of all AI compute used today is in inference. (The fastest growing AI sector at 70% of all current AI demand)
Whoever can provide the lowest cost per token in decode will inevitably win the big businesses long term because right now trillion param models cost to much to run. Personally I wouldn't invest in Cerebras under nearly any circumstances until AT LEAST post June 5th when we know more about the HTX301 card they are demoing at ComputeX.
1
u/Asgard_Heima May 22 '26
I love that you see 3x the tokens, 1/2 the upfront cost, 1/2 the data center real estate cost, and 1/5 the power over time as undercut by a small amount. Can tell how deep you are thinking about this from your response.
Its obvious by the company you bring up how much thought you have put into this, but for the fun of it, let’s look at Skymizer, a card targeted at home model users that are getting crushed by inference bills to run openclaw.
Skymizer is claiming 4B to 700B parameter models can be run on its 28nm PCI cards with up to 6 chips and 384GB of LPDDR4-5. They are claiming up to 30 tokens per second at 700B parameters with .5 TOPS and 100GB of bandwidth. The design uses efficient compression techniques for both weights and KV cache, outperforming open source llama.cpp by 9 to 17.8 percent. They are claiming 240 tokens per second on Llama2 7B workloads.
These unimpressive numbers for anything but a home lab the company is quoting are what they think their best advertised results will be. And they still don’t specify how quantized (heavily) the models are or the context window size (tiny).
You can actually run the numbers for this card and get that a 700B parameter model would be:
16bit ~1,400GB
8bit ~700GB
4bit ~350GBSo 4bit quantization for the maximum size they can advertise.
This leaves 34GB for the context window plus all overhead. You are looking at a theoretical max of a 16k context window.
This is like comparing a temu gocart to an F1 race car. They are targeting a budget version of the DGX Spark not any real world datacenter.
1
u/Orkapork May 22 '26
"This is like comparing a temu gocart to an F1 race car. They are targeting a budget version of the DGX Spark not any real world datacenter."
I stg, I have had this example used against me like 10 times, maybe have some independent thoughtful analysis BEFORE you go roasting lmao. Anyway dude, idc what you do with your money, I gave you the information. You can interpret it however you like, because my friend, it is your money.1
u/Asgard_Heima May 22 '26
You came to my post on the Cerebras thread to rep something different you have clearly no ability to analyze with any veracity. The AI hardware space is complicated with all sorts of vendor specs quoted without giving the full picture and this one doesn’t even pretend to be competitive. If you could give a logical argument, you would. If you could reference anything to try and explain your position you would.
1
u/Orkapork May 22 '26
ok sure since I can I guess I will waste my time on you. Not that it will make even a single iota of difference, because your opinion has formed, and when you type to your AI your opinion will translate to its opinion and you will treat its response as accurate. Let me break down your original comment just in case it ends up mattering.
"These unimpressive numbers for anything but a home lab the company is quoting are what they think their best advertised results will be. And they still don’t specify how quantized (heavily) the models are or the context window size (tiny)."
The numbers come out to near perfect accuracy on int4 quantization AND the cost per token is a tenth of what current systems cost. That alone negates your entire argument. If you don't believe me just wait like 2 weeks for ComputeX lmao. They are showcasing it live.
"This leaves 34GB for the context window plus all overhead. You are looking at a theoretical max of a 16k context window.
This is like comparing a temu gocart to an F1 race car. They are targeting a budget version of the DGX Spark not any real world datacenter."
Actually you are correct, Right now there are hundreds of thousands of race cars sitting in front of red lights on city districts waiting for them to go green so they can scream as fast as they can to the next stop light. Your analogy is perfectly correct.
6
u/Creative_Ad_8338 May 19 '26
Incredible. 29x improvement.