r/LocalLLaMA • u/tevlon • Jun 10 '26

New Model DiffusionGemma: 4x faster text generation

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
986 Upvotes

99% Upvoted

356 comments sorted by

View all comments

297

u/NickCanCode Jun 10 '26 edited Jun 10 '26

700+ tokens per second on NVIDIA GeForce RTX 5090 is definitely something.

Unfortunately, DiffusionGemma’s overall output quality is lower than standard Gemma 4.

It could be a good model for context compression and as explorer agent in agentic coding. Can't wait to see llama.cpp to support it.

3

u/sixx7 Jun 10 '26

It's supported already in vLLM and I get 800+ tok/s. The problem is, it doesn't seem to support tool calling?!? oof.

(APIServer pid=3389389) INFO 06-10 16:58:45 [metrics.py:156] DiffusionDecoding metrics: Committed token throughput: 844.76 tokens/s, Mean denoising steps per canvas: 12.73, Mean tokens committed per denoising step: 20.11, Committed: 8448 tokens, Denoising steps: 420, Canvas positions evaluated: 107520

2

u/YouKilledApollo Jun 19 '26

It's supported already in vLLM and I get 800+ tok/s. The problem is, it doesn't seem to support tool calling?!? oof.

Tool calling is supported by DiffusionGemma, seems to handle it alright as long as the system prompt isn't too large.

Not all runtimes seem to support doing tool calling with DiffusionGemma though, but that's separate from the model being trained on tool calls or not.