Chimera: a Chinchilla scaling law for hybrid diffusion transformers

The problem with full attention at video scale

Every team building image and video diffusion models runs into the same wall eventually: full attention costs O(N^2) in the number of tokens, and video tokens accumulate fast. A 30-second clip at reasonable resolution can be an order of magnitude more tokens than a single image. You either cap your context, pay a compute bill that scales quadratically, or reach for a linear-attention approximation and accept whatever quality loss comes with it. None of these are satisfying answers, and none of them tell you how to scale the architecture you land on once you've picked one.

Chimera, from a team including authors at Adobe, takes on both halves of that problem at once: a hybrid backbone designed to make the attention-cost tradeoff less punishing, and a scaling recipe that tells you how to size and train it. That second piece is the part I think is underappreciated. Plenty of papers propose a new hybrid block. Far fewer propose how to fit a Chinchilla-style compute-optimal law to something that isn't a uniform stack of identical transformer layers.

The architecture: three ways of moving information, one stream

Chimera processes text, image, and video tokens as a single raster-ordered stream, with no positional embeddings. Inside each block, three mechanisms handle information flow at different scales:

• Kimi Delta Attention (KDA), a linear-attention variant, does the heavy lifting for long-context state tracking at O(N) complexity. This is what makes long video sequences tractable at all.
• Multi-head Latent Attention (MLA) is interleaved between KDA layers to give the model direct global interaction, a channel for information to move across the whole sequence without going through the state-tracking bottleneck of the linear path.
• Modality-aware short convolutions handle local spatiotemporal context, the fine-grained, nearby-token structure that attention (linear or full) is comparatively inefficient at capturing.

On top of that, sparse Mixture-of-Experts layers expand model capacity while keeping activated compute in check. The final model the authors train has 11B total parameters with only 2B activated per forward pass, and the gap between those two numbers is most of the efficiency story.

None of these components is individually novel: linear attention, latent attention, and MoE all have a research lineage. What's being tested here is whether stacking them into one heterogeneous block, rather than picking one mechanism uniformly, is worth the added design complexity.

HeteroP: transferring hyperparameters across a non-uniform architecture

Here is the harder problem the paper actually solves. Standard hyperparameter-transfer techniques such as muP assume you are scaling a roughly homogeneous transformer: same attention type, same block shape, just more of it. Chimera's blocks are not uniform. KDA, MLA, convolutions, and MoE experts all have different fan-in structure and sit at different depths. Hyperparameters tuned on a small version of this architecture do not transfer cleanly to a larger one if you treat every module the same way.

The paper's answer is HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and its position in the model's depth. Instead of one global rule, each module type gets its own transfer rule derived from its actual role in the computation graph. The payoff is a consistently tuned family of models across scales, which is the precondition for fitting a scaling law in the first place. You cannot fit a reliable compute-optimal curve on models that are each undertuned or overtuned relative to their size; the noise from bad hyperparameters swamps the signal you are trying to measure.

With that family in hand, the authors fit Chinchilla-style compute-optimal laws across three axes: activated model size, training-token count, and the ratio of image to video data in pretraining. That last axis is specific to this domain, there is no equivalent in text-only scaling laws, and it is a genuinely useful knob for any team mixing image and video pretraining data.

What the fitted laws say, and what it costs

Guided by these laws, the authors train an 11B-parameter Chimera with 2B activated parameters, and report three results.

First, compute efficiency. Measured by pretraining diffusion loss, the dense backbone alone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline. The complete system, dense backbone plus MoE, reaches 7.3x. That is a large gap between what the attention redesign buys you and what the full system buys you, and it is worth sitting with: most of the win comes from combining sparsity with the hybrid attention, not from the attention change alone.

Second, length extrapolation. Without any length-specific fine-tuning, Chimera trained on 5-second clips extrapolates zero-shot to 30-second videos, with only 6.5% FID degradation in the final five seconds. For a linear-attention-heavy architecture, holding up this well six times past training length is a meaningful result. It is evidence the KDA/MLA split is doing real state-tracking work rather than just approximating full attention over a fixed window.

Third, compute allocation. The fitted laws show compute-optimal image pretraining splits nearly evenly between activated model size and training-token count. Video pretraining modestly favors model size at higher budgets. If you are planning a training run and have been using text-model scaling intuition, roughly Chinchilla-optimal token-to-parameter ratios, to guide image and video budget splits, this result says video data specifically wants relatively more parameters as budget grows.

Why this matters beyond the benchmark numbers

The headline efficiency numbers will get cited, but the transferable lesson is the HeteroP methodology itself. Any team building a heterogeneous architecture, mixing attention types, adding MoE, bolting on modality-specific modules, hits the same wall: hyperparameter transfer techniques built for uniform transformers do not apply cleanly, and without a fix, you either waste compute retuning at every scale or draw scaling conclusions from a poorly tuned model family. A module-wise transfer rule keyed to fan-in and depth is a reusable pattern well beyond diffusion models, and I would expect to see variants of it show up anywhere hybrid architectures are being scaled seriously rather than as one-off experiments.

The practical takeaway for teams working on long-context generation: the compute-optimal ratio is not the same for image and video data, and it changes with budget. That is a concrete input into planning the next training run, not just an interesting research footnote.