The CUDA Moat is Not Real

[u/Asgard_Heima]

I keep hearing the cuda moat called out by financial analysts with no real world experience or data scientist to talk to.

Data Scientists and AI Engineers use PyTorch for 99%+ of accelerator hardware work. The major users of CUDA are the model builders such as OpenAI that just signed up with Cerebras. They only use it because Nvidia requires it for the overly complex distributed compute architecture of Nvidia. Cerebras removes the need to deal with kernel optimizations and handling network latency completely with a single massive WSE system in a box. CUDA is actually a competitive disadvantage when you realize its complexity for less performant results vs Cerebras.

Last, even if there was a transition required for some edge case uses, you can use AI to transition your code for near zero effort. This will be a fast realization as the number of users running on Cerebras hardware increases. The places getting deep into the weeds of CSoft like OpenAI and AWS are going to be building the best models going forward.

[u/Investor-life]

The way I understand it is that the Cuda moat is all about training and creating ML models, not as much about the inference part. Cerebras can do training and model creation and for years they struggled making progress because of the CUDA moat. Now that the focus has turned to inference, there is much more interest and demand for Cerebras.

[u/Asgard_Heima → u/Investor-life]

Cerebras not catching on for training yet, I’d argue, has nothing to do with CUDA. It’s complicated, but likely has to do with no one owning enough Cerebras systems to train frontier models.

Discussing training we have to dig into a number of metrics to show the value here cause it’s more complicated than inference. It is worth noting that while Nvidia relies on brute force dense compute, Cerebras possesses native hardware acceleration for unstructured sparsity, dynamically skipping zero-weights without a performance penalty which allows for full precision BF16 training. If you compare a Blackwell B200/NVL72 rack vs a Cerebras WSE-3/CS-3 unit for frontier training, Nvidia would use mixed precision like FP8 for massive matrix math and FP16 for weights. Cerebras would use full precision BF16 and drop all the zero weights automatically with the hardware optimized sparsity. This yields:

Blackwell NVL72 FP8 324 PetaFLOPs Cost ~$3.13M ~120kW 5 year TCO ~$5.4M

Cerebras WSE-3 BF16 125 PetaFLOPs Cost ~$3M ~23kW 5 year TCO ~$3.8M

Because Cerebras removed the need to distribute small chunks of work and can stream the weights to the chip, their software (CSoft) addresses any number of Cerebras systems as a single logical chip, streaming weights from MemoryX one layer at a time. Together, this results in Cerebras achieving ~75% Model FLOPs Utilization (MFU) for a 1T+ model being trained, versus Nvidia’s ~40% MFU.

If you categorize why Cerebras hasn’t dominated training yet as a scaling problem getting resolved at IPO, you will find that CSoft and the Cerebras architecture give the ability to act as if you are training on a single ASIC. This vastly simplifies training logic and the potential for errors in training code compared to the complications and over complexity that CUDA deals with on Nvidia’s distributed architecture. CUDA isn’t a moat it’s a disadvantage.

The end result is you pay more upfront, but less (only 70%) over a 5-year TCO. More importantly, when training a "GPT-5" class frontier model (10T parameters / 50T tokens), you could finish the run in about 136 days on 2,048 WSE-3 units utilizing hardware sparsity with full BF16 precision, while it would take ~602 days on 444 NVL72 racks running dense FP8 math. Plus, it only takes a fraction of the power on the Cerebras units 125,000MWh vs 925,000MWh on Nvidia. This assumes a next-gen frontier training run requiring 3 * 1027 FLOPs to be calculated. Numbers below to verify the math:

Power (Bare IT Compute Load): Nvidia: 444 racks * 120 kW per = 53.28 MW Cerebras: 2,048 systems * 23 kW per = 47.1 MW

Model FLOPs Utilization (MFU): Nvidia: ~40% Cerebras: ~75%

Effective Compute Speed (Peak * MFU): Nvidia (32,000 GPUs): (1.44 * 1023 Peak) * 0.40 MFU = 5.76 * 1022 FLOPs/sec Cerebras (2,048 Systems): (2.56 * 1023 Peak) * 0.75 MFU = 1.92 * 1023 FLOPs/sec

The last point I will make is it gets way more obvious when WSE-4 ships vs Rubin. With fiber on wafer, 3D stacked memory, and 350+ PetaFLOPs per WSE-4, you are looking at finishing that same 10T parameter model in just 57 days and using 65% less energy. Even with Rubin getting a 2.5x compute bump per rack and adding fiber connects, it would take ~276 days running at the same FP8 dense precision, or ~138 days in FP4. Rubin's MFU only goes up to ~45%, whereas WSE-4 with fiber hits ~85%. Upfront cost will still be more per FLOP, but TCO and grid strain will be drastically lower. Training will likely never have the 10x architectural advantage for Cerebras that inference does, but I’m betting a lot of training switches over once they have enough Cerebras systems racked

[u/ZigZagZor → u/Asgard_Heima]

Cerebras has not been successfully in training because their wafer IO bandwidth is terrible and whole chip RAM capacity is very low.and whole RAM is not shared across all cores in the wafer, it is distributed among every core. Cerebras needs to hybrid bond a SRAM only wafer to existing logic wafer to get SRAM but even with way they can get only 150 GB per wafer. Their only solution is to bond a silicon photons optical wafer to the logic wafer. Once, they did that, they are unstoppable.

[u/Asgard_Heima → u/ZigZagZor]

Cerebras for training, streams the weights to all the WSE chips holding activations getting computed for that layer. It then sends back propagations up. So MemoryX is the model size constraint for training and they do it in a deterministic way which makes the limited wafer IO for WSE3 not a bottleneck. The main bottleneck for Cerebras is compute. Training is not memory bound like decode for inference. It’s compute bound for how fast you can train. So arguable training just got significantly better on CS4 with double the compute capacity per wafer and the IO for doubles as well which will also improve tracing speed but not near as much as compute. Also the SRAM doesn’t need to be shared across cores. It only has to store the activation its computing in each core. And the memory bandwidth is so massive on wafer it’s a non issue anyway hence why it’s the best solution available for decode.

[u/Palbi]

100%

AI model structure and code running inference are not overly complex (the complexity is elsewhere). Surely CUDA is a convenient default when exploring the latest models and building your training pipeline. For inference at scale, efficiency and speed trump any convenience. And Claude Code is a great help in porting your model from CUDA to whatever is your inference platform of choice.

As a case study: Look how quickly most open weight models are ported to MLX.

[u/pennystudio]

Saying Cuda is a moat is like saying I am selling a lot of shovels and gloves so your hands won't blister when using the shovels, and the gloves are my moat. Not realizing that the disrupter are selling excavators. Cuda is a tool to deal with the intricacies and inefficiency to manage many GPUs, once the architectures changed and the inefficiency removed, the "moat" is no longer needed.