Lossless speedups in LLM decoding, without a draft model

Every autoregressive LLM I've put into production shares the same tax: one forward pass buys you exactly one token. Throughput at scale is mostly a fight against that constraint — batching, speculative decoding, KV-cache tricks, quantization — all of it working around a sequential bottleneck nobody has removed. A paper posted this week, Unlocking Lossless Speedups in LLMs via Discrete Diffusion (Sahoo et al., arXiv:2609.04010), goes after the bottleneck directly, and the mechanism is unusual enough that I want to walk through it.

The core move: split the weights

The authors decouple a model's parameters into two sets. The AR weights are trained exactly as usual, with next-token prediction (NTP) — nothing about the base model's training changes. On top of that, they train a separate, lightweight set of diffusion weights whose job is to draw several tokens at once from the same distribution the AR model already defines, rather than approximate a different one. That second phase is called Diffusion Distillation, and per the abstract it adds negligible overhead to an existing LLM training pipeline — this isn't a from-scratch retraining commitment. Crucially, the paper reports you can apply this to models trained from scratch or bolt it onto existing open-weight AR LLMs, which is the detail that makes it interesting to anyone with a model already in production rather than a research budget for a new pretraining run.

This is a different bet than diffusion LLMs (d-LLMs) like Mercury or DiffusionGemma, which replace the AR objective outright and have historically traded away some of the quality autoregressive training gets you. Here the AR model stays the model. Diffusion is only the mechanism for proposing multiple tokens per step; it never gets to redefine what a correct continuation looks like.

Ψ-Spec: exactness without a draft model

The sampler that ties this together is called Ψ-Spec, and the claim that matters most is lossless. Speculative decoding — the dominant technique for multi-token generation today — gets its speed from a small draft model proposing tokens that a larger model then verifies, accepting or rejecting them so the output distribution stays exact. It works, but it means running and maintaining a second model, tuned to draft well for the first.

Ψ-Spec drops that second model. The diffusion weights, trained directly against the AR model's own distribution rather than as an independent proposer, generate the candidate tokens; the sampler's job is to accept them in a way that reproduces exactly what the AR model would have produced token-by-token. No separate draft model to serve, version, or keep in sync — the acceleration lives inside the model you already deployed. The paper also describes Ψ-Spec as enabling inference-time scaling at a fixed context length, meaning the multi-token trick isn't just a decode-speed knob but something you can lean on harder when you want more compute spent per response without extending the sequence.

The numbers

The resulting models are called Uno. Per the paper, Uno beats leading speculative-decoding methods on throughput at every batch size the authors evaluated, and delivers up to 3x speedup over the base AR model — including at the largest batch size the test device could support, which is exactly the regime where speculative decoding's gains tend to shrink because verification cost scales with batch size too. That's the number worth sitting with: this isn't a small-batch party trick, it holds where production traffic actually lives.

The quality claim is just as notable. An 8B Uno model is reported to outperform the leading open d-LLM — the 26B DiffusionGemma — and the proprietary Mercury 2, across every benchmark evaluated in agentic tool use, coding, and long-context reasoning. If that holds up under independent replication, it's a direct answer to the standard objection against diffusion-style generation: that you buy parallelism by giving up something in reasoning-heavy or long-context tasks. An 8B model beating a 26B one on the tasks that are hardest for non-autoregressive methods is a strong claim, and the honest caveat is that leading and evaluated are the authors' framing — I'd want to see this benchmarked by people with no stake in the result before treating it as settled.

Why I'd read this closely if I were serving models at scale

The part I find most practically relevant isn't the speedup number, it's the deployment shape. Speculative decoding asks you to operationalize a second model — pick a draft model, keep it compatible with the target model across updates, and accept that its usefulness varies by workload. Ψ-Spec's pitch is that the acceleration is a property of the model itself, added through a cheap distillation pass on top of a model you already trained or already run. That's a meaningfully smaller operational surface, and it's compatible with applying it to open-weight models you didn't train yourself.

What I'd want answered before betting infrastructure on it: how the 3x figure moves across model sizes and hardware beyond what's reported, how much the Diffusion Distillation phase actually costs at the scale of a large production checkpoint rather than negligible in relative terms, and whether the benchmark suite generalizes to the messier, tool-heavy agentic workloads most of us are actually running rather than the evaluated set. The code and checkpoints are released, which means those questions are answerable rather than speculative. That's the right way to make a claim like this one.