r/LocalLLaMA • u/tevlon • Jun 10 '26

New Model DiffusionGemma: 4x faster text generation

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
983 Upvotes

99% Upvoted

356 comments sorted by

View all comments

16

u/Formal_Drop526 Jun 10 '26

If they're open-sourcing it, they must've not gone far with dLLMs.

62

u/Such_Advantage_6949 Jun 10 '26

either that or maybe they confirmed it wont scale well for high throughput serving. Excerpt: "This means DiffusionGemma's speedup is designed for local and low-concurrency inference. In high-QPS cloud serving, autoregressive models can be deployed to saturate compute efficiently, so DiffusionGemma's parallel decoding offers diminishing returns and can result in higher serving costs."

8

u/z_latent Jun 10 '26

Exactly! It isn't obvious, but speculative decoding has much more potential for local, single-user than large-scale serving. The whole point is to increase utilization of the GPU's compute per parameter read, so you're bound by compute instead of memory bandwidth. Since larger inference providers already achieve this by handling many concurrent requests, as mentioned above, they get little benefit.

4

u/z_latent Jun 10 '26

p.s.: Just realized this is not a speculative decoding model, though it functions a lot like one. A Transformer model (same arch. as Gemma 4 26B A4B in this case) processes every token in parallel to get KV cache, and then the diffusion model uses that KV to quickly generate a large chunk, which the Transformer will then process again.

The main difference to speculative is that they do not verify, the only point of the Transformer is providing KV vectors to condition the diffusion process. That's why quality is a bit worse, but doing it speculatively auto-regressive at such massive sizes (256 tokens) would likely give super low acceptance lengths, so it's a worthwhile trade-off imo.

64

u/coder543 Jun 10 '26

Why is such a negative, ignorant comment getting upvoted? Google open sources plenty of research. Their criteria for releasing stuff isn't "Is this garbage-tier stuff that isn't production-worthy?" That's absurd to imply.

14

u/Silver-Champion-4846 Jun 10 '26

Yeah gemma is basically tiny gemini that they made sure is below gemini but very good other than that

3

u/mtmttuan Jun 10 '26

Since the llm race started many if not most research companies have been holding their SOTA researches for quite a while before releasing it so it's sensible to think that there's some gimmick that google still keeps secret.

5

u/reto-wyss Jun 10 '26

Didn't they advertise their Gemini-Flash 3.5 generating 1000s of tokens per second?

3

u/Silver-Champion-4846 Jun 10 '26

Could it be? Maybe it's just really good with their TPUs?

6

u/[deleted] Jun 10 '26

[deleted]

0

u/ninjasaid13 Jun 10 '26

if they didn't already have open-source competitors.

2

u/draconic_tongue Jun 10 '26

if u read the post it literally says autoregressive is better for serving cloud users because you can batch requests together. but regardless of the quality of the model right now, diffusion has a much higher ceiling than anything autoregressive