DiffusionGemma: fine-tuning Gemma 4 into a 1,500-token-per-second diffusion decoder

Google's Gemma team just published something that inverts an assumption most production LLM stacks treat as fixed: that generation has to happen one token per forward pass. In the DiffusionGemma Technical Report, they take a real, deployed-scale mixture-of-experts model — Gemma 4, 3.8B active parameters out of 25.2B total — and fine-tune it into a discrete-diffusion variant that refines blocks of 256 tokens in parallel instead of decoding sequentially. Averaged across their evaluation suite, it generates around 20 tokens per forward pass and sustains roughly 1,500 output tokens per second on a single H100. It gets there using under 10% of the original model's total training token budget, and it keeps thinking mode, multimodal inputs, and long context intact.

That combination — production-scale base model, fine-tuning rather than pretraining, and a genuine architectural switch in how tokens get generated — is what makes this worth a close read rather than a skim of the abstract.

The bottleneck this attacks

Autoregressive decoding has one hard constraint: token N+1 needs token N. Every optimization built on top of that — KV caching, continuous batching, speculative decoding — works around that sequential dependency rather than removing it. Speculative decoding, the current best answer, drafts several tokens with a cheap model and verifies them in parallel with the target model, but rejected drafts get thrown away and the acceptance rate caps how much parallelism you actually recover. DiffusionGemma's reported throughput is claimed to beat state-of-the-art speculative decoding, which is the comparison that matters if speculative decoding has been your only lever for cutting generation latency.

Blocks, not tokens

Instead of emitting tokens left to right, DiffusionGemma treats a chunk of up to 256 tokens as a single denoising target. It starts from a masked or noised block and iteratively refines all positions in that block together, converging over a handful of forward passes rather than one pass per token. That's the structural reason the token-per-forward-pass ratio can land around 20:1 — a small number of iterative refinement steps produces a much larger number of finished tokens, and each step is a batched operation across the whole block rather than a single-token decode.

The tradeoff diffusion models have always carried is quality: earlier discrete-diffusion language models lagged autoregressive baselines on coherence and instruction-following, which is part of why this technique hasn't shown up in a widely deployed production model until now. The report frames DiffusionGemma as establishing a new Pareto frontier between generation speed and model capability — the claim isn't that diffusion is now categorically better, it's that the speed-versus-quality curve moved.

Fine-tuning a diffusion model out of an autoregressive one

The part I find more interesting than the throughput number is how they got there. They didn't train a diffusion model from scratch — they started from the already-trained Gemma 4 checkpoint and ran a two-stage, compute-efficient pipeline that uses under 10% of the original AR model's training token budget.

Stage one is supervised fine-tuning to teach bidirectional denoising: the model learns to predict masked positions using context on both sides, a different objective than the causal, left-to-right prediction it was pretrained on. Stage two combines reinforcement learning with sampler distillation, jointly optimizing generation quality and inference efficiency rather than treating those as separate passes. That second stage does double duty — improving what the model outputs and how cheaply it can output it — inside the same optimization loop.

The practical implication is that you're not paying pretraining-scale compute to get a diffusion model with the language competence of a mature checkpoint. You're renting capability that's already baked into Gemma 4 and reshaping how it gets decoded.

What survives the conversion

Bidirectional denoising is a real shift away from causal attention, so it's fair to ask what breaks. The report's answer: not much. DiffusionGemma keeps thinking mode, multimodal inputs, and long-context support from the base model. It also remains capable of standard autoregressive generation with only minor performance degradation — the fine-tuning didn't overwrite the AR pathway so much as add a diffusion one alongside it. The authors call this out explicitly as a path toward hybrid diffusion-AR decoding, where a system could choose the decoding mode per request: diffusion for throughput-bound generation, AR fallback where diffusion's block structure is a worse fit, such as long chains of tight sequential reasoning where token-by-token verification matters more than raw speed.

Why I'd pay attention to this if I were building on it

Generation latency is usually the least controllable line item in an agentic system — the model call sits in the middle of a loop, and everything else, tool calls, retries, orchestration, inherits its tail latency. A 1,500 tokens-per-second, single-GPU number, if it holds up outside the paper's own eval suite, changes the economics of anything that generates long outputs under a latency budget: long-context summarization, code generation, multi-turn agent traces where the model has to emit verbose intermediate reasoning before it can act.

The more transferable result, though, is the recipe, not the specific checkpoint. If a fine-tune costing under 10% of a pretraining budget reliably converts an autoregressive checkpoint into a fast diffusion decoder while preserving thinking mode and multimodality, that's a technique other model families could adopt without eating a full pretraining run. I'd want to see the composition of that eval suite before assuming the 20-tokens-per-forward-pass average generalizes to my own workloads, and I'd want throughput numbers reproduced outside a single self-reported H100 benchmark before moving a production path onto it. But the direction — decoupling how good a model is from how it has to decode — is worth tracking regardless of whether DiffusionGemma specifically ships.