DiffusionGemma: 4x faster text, and where the speedup actually lands

Last week Google DeepMind released DiffusionGemma, an experimental open model that generates text by diffusion rather than the usual one-token-at-a-time prediction. It is built on the Gemma 4 family, ships under an Apache 2.0 license, and the headline is up to 4x faster generation on GPUs — over 1,000 tokens per second on a single NVIDIA H100, and 700+ on a consumer RTX 5090.

That speed comes from a different decoding strategy, and the strategy comes with a specific tradeoff. Both are worth understanding before you reach for it.

Typewriter versus printing press

A standard autoregressive language model generates one token at a time, each conditioned on the tokens before it. For a single local user, that is the wrong shape of work for a modern GPU: the model spends most of its time waiting on memory bandwidth to fetch weights for the next token, and the accelerator's compute sits mostly idle. DeepMind's own framing is that the decode bottleneck is memory-bandwidth, not compute.

DiffusionGemma reverses that. Instead of emitting words sequentially, it drafts an entire 256-token block in a single forward pass and then refines it over several passes. Giving the GPU one large chunk of arithmetic at once — rather than a long string of tiny dependent steps — moves the bottleneck from memory bandwidth to compute, which is exactly what the hardware is good at. The blog post's metaphor is apt: a typewriter that types one key at a time becomes a printing press that stamps the whole block.

How the model is built

DiffusionGemma is a 26B-parameter Mixture-of-Experts model that activates only 3.8B parameters per forward pass, so quantized it fits inside the 18GB VRAM of a high-end consumer card. The decode loop works the way image diffusion does: the model starts from a canvas of random placeholder tokens, makes repeated passes that lock in the tokens it is confident about, uses those as context to fix the rest, and converges on the final text.

The consequence I find most interesting is architectural, not numerical. Because the whole block is visible during generation, the model uses bi-directional attention — every token can attend to every other, including ones "after" it. That is a genuine fit for tasks that are not naturally left-to-right: in-line editing, code infilling, structured layouts that must close cleanly, even non-text sequences like amino acids. DeepMind cites an Unsloth fine-tune that solves Sudoku, a problem autoregressive models handle poorly precisely because each cell depends on cells that come later.

Where the speedup actually lands

This is the part I would underline before anyone swaps it into a stack. The 4x win is for local, low-concurrency inference — a single user on a single accelerator at low-to-medium batch size. In high-QPS cloud serving, autoregressive models already batch thousands of requests together and saturate compute on their own, so diffusion's parallel decode offers diminishing returns and can actually raise serving cost. DeepMind also notes that unified-memory machines like Apple Silicon, with a lower compute-to-bandwidth ratio, may not see the same gain.

And there is a quality line drawn explicitly: DiffusionGemma's output quality is below standard autoregressive Gemma 4, and the recommendation for quality-critical work is to keep using the autoregressive model. This is an experimental release aimed at researchers and developers building interactive local workflows, not a drop-in upgrade.

My take

The number that travels is "4x," but the durable idea is the reframing of where the bottleneck lives. Autoregressive decoding is a memory-bandwidth problem dressed up as a compute one; diffusion trades that for a compute problem the hardware can chew through, and it pays off precisely in the regime — one user, one GPU — where batching can't rescue you. That is a real gap in the open-weights ecosystem, and shipping it Apache 2.0 at a size that runs on a desktop card is the part that matters most.

I would not reach for it where output quality is the constraint; the honest quality caveat means it earns its place through latency and through tasks that are inherently non-sequential, then through fine-tuning for the specific job. Weights and a developer guide are on Hugging Face, with serving through MLX, vLLM, and Transformers and llama.cpp support said to be arriving. If you build anything interactive that runs locally, it is worth an afternoon — with clear eyes about what you are trading for the speed.