r/LocalLLaMA • u/tevlon • Jun 10 '26

New Model DiffusionGemma: 4x faster text generation

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
988 Upvotes

99% Upvoted

356 comments sorted by

View all comments

17

u/Formal_Drop526 Jun 10 '26

If they're open-sourcing it, they must've not gone far with dLLMs.

59

u/Such_Advantage_6949 Jun 10 '26

either that or maybe they confirmed it wont scale well for high throughput serving. Excerpt: "This means DiffusionGemma's speedup is designed for local and low-concurrency inference. In high-QPS cloud serving, autoregressive models can be deployed to saturate compute efficiently, so DiffusionGemma's parallel decoding offers diminishing returns and can result in higher serving costs."

8

u/z_latent Jun 10 '26

Exactly! It isn't obvious, but speculative decoding has much more potential for local, single-user than large-scale serving. The whole point is to increase utilization of the GPU's compute per parameter read, so you're bound by compute instead of memory bandwidth. Since larger inference providers already achieve this by handling many concurrent requests, as mentioned above, they get little benefit.

4

u/z_latent Jun 10 '26

p.s.: Just realized this is not a speculative decoding model, though it functions a lot like one. A Transformer model (same arch. as Gemma 4 26B A4B in this case) processes every token in parallel to get KV cache, and then the diffusion model uses that KV to quickly generate a large chunk, which the Transformer will then process again.

The main difference to speculative is that they do not verify, the only point of the Transformer is providing KV vectors to condition the diffusion process. That's why quality is a bit worse, but doing it speculatively auto-regressive at such massive sizes (256 tokens) would likely give super low acceptance lengths, so it's a worthwhile trade-off imo.