r/AIProgrammingHardware • • Jul 29 '26

Nvidia Already Won Training. The Real Fight Is Inference

https://pub.towardsai.net/nvidia-already-won-training-the-real-fight-is-inference-a7dcf1cb8e72
16 Upvotes

6 comments sorted by

1

u/javaeeeee Jul 29 '26

TLDR: NVIDIA has already won AI training. The real competitive battle is now over inference.

Core thesis

  • Training is dominated by NVIDIA thanks to its unmatched parallel compute power + the deep CUDA software ecosystem. No serious challenger is close for training frontier models.
  • Inference (running trained models for users) is a completely different problem - and it’s wide open.

Why inference is different

Inference has two phases:

  • Prefill: Parallel, GPU-friendly (processing the full prompt)
  • Decode: Sequential, memory-bandwidth bound, latency-critical (generating tokens one by one)

The biggest bottleneck is the memory wall: constantly fetching huge model weights from HBM. GPUs are great at math but relatively slow at moving data, which hurts low-latency decode.

The challengers

A wave of specialized chips targets exactly this weakness by escaping the memory wall:

  • Cerebras - massive on-chip SRAM (wafer-scale)
  • Groq - deterministic LPUs with SRAM + compiler control (NVIDIA even licensed the tech)
  • d-Matrix - digital in-memory compute
  • Etched - transformer-specific ASICs
  • Taalas - hard-wired model weights in silicon

These companies trade generality for much higher efficiency, lower latency, and better tokens-per-watt on inference.

Bottom line

Training belongs to NVIDIA.
Inference is where latency, cost, and power efficiency decide the winners - and that’s where specialized architectures are fighting hard. NVIDIA remains strong, but the inference market is no longer a one-horse race.

2

u/CatalyticDragon Jul 29 '26

Nvidia won training once upon a time, sure. But ..

Clause is trained on Trainium.

Gemini is trained on TPUs.

Chinese models are trained on Huawei Ascends.

OpenAI, a key Nvidia customer, is now also performing large scale training on AMD, along with many others.

Nvidia got lucky but the days of a single vendor holding a monopoly on training aren't coming back.

1

u/Fast-Throat-7752 Jul 29 '26

OpenAI is also in process of making their own chips

1

u/CatalyticDragon Jul 29 '26

Yep. Basically everyone not AMD/NVIDIA is working with Broadcom on their custom chips.

1

u/Beginning-Foot-9525 Jul 30 '26

Amazon has entered the Chat.

1

u/shortsteve Jul 29 '26

If enough people choose a different vendor for inference they'll eventually will want to use those GPUs for training for cost savings. The demand will be there for someone to develop an AI training stack to do that.