MoE-ViE: making mixture-of-experts actually pay off in vision encoders

Mixture-of-experts is the reason large language models keep scaling capacity without scaling inference cost in lockstep: route each token to a handful of specialists out of a much larger pool, and you get more parameters without more compute per token. Vision encoders haven't gotten the same deal — prior attempts to bring MoE into CLIP-style encoders existed, but none had closed the gap to dense state-of-the-art. MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding, a Meta paper accepted to ECCV 2026, closes it — and how the authors get there is a lesson in how much the details of an MoE design matter, not just the decision to use one.

Fine-grained experts, not a coarse swap

The obvious way to add MoE to a vision transformer is to replace each dense MLP block with N experts that each match the original MLP's capacity, and route tokens among them. The paper shows this "conventional MoE" gets you only modest gains. At ViT-B/32 scale, average classification accuracy goes from 55.8 (dense) to 59.9 (conventional MoE) — a real but limited jump.

Instead, the authors use a fine-grained design: shrink each expert's hidden width so many more fit under the same compute budget, then activate only a fraction per token. Their configuration uses 32 experts per MoE layer, each a quarter the width of the corresponding dense MLP, with only 1/8 or 1/4 active at once. The reasoning is that visual features are heterogeneous — color, shape, spatial layout, domain — and many narrow specialists capture that variety better than a few large ones, mirroring what fine-grained MoE has already shown in LLMs. Under the same ViT-B/32 budget, this pushes average classification to 63.2, more than triple the conventional design's gain. The pattern holds at ViT-L/16 scale too (78.0 dense to 78.5 conventional to 79.6 fine-grained). MoE is applied only to the vision tower — the text tower stays dense — and the first transformer block is left dense as well, since it captures low-level visual information that didn't benefit from specialization.

On top of granularity, they swap the usual softmax gate for a sigmoid gate, renormalized after top-k selection, to avoid experts competing for probability mass, and add a small set of shared experts that are always active alongside the routed ones — a persistent pathway for global context, while the routed experts handle token-specific specialization.

A steadier loss-free balancing rule

MoE training has a chronic problem: without some mechanism to spread load, a few experts get most of the tokens and the rest go unused. The now-standard fix, loss-free balancing, adds a per-expert bias to the routing logits and adjusts it based on observed token load rather than an auxiliary loss term that fights the main training objective. The original version nudges each bias by a fixed step in the direction of imbalance — but a constant-magnitude correction regardless of how imbalanced things actually are causes oscillation once routing is close to balanced. MoE-ViE replaces that fixed step with one scaled by the z-score of the load imbalance, so the correction shrinks as balance improves. In their ablation, this magnitude-aware version outperforms the original loss-free method, an entropy-based auxiliary loss, and an importance-and-load auxiliary loss on every benchmark tested.

Sparsity is only fast if the kernel is fast

MoE's efficiency argument is theoretical until the implementation delivers it. A naive PyTorch implementation that loops over experts underuses GPUs: many small matrix multiplies keep arithmetic intensity low, dynamic routing forces CPU-GPU synchronization to size per-expert workloads, and larger intermediate tensors make memory bandwidth the real bottleneck. The authors built a Triton kernel around two optimizations — grouped GEMM, which batches all expert matrix multiplies into a single GPU task instead of many small ones, and kernel fusion, which folds the matmul and activation function into one pass to cut round-trips to memory. At a batch size of 128 and 576 tokens, their MoE-ViE-H model runs in 545 ms with the optimized kernel versus 1,307 ms vanilla — over 2.5x faster, consistently across batch sizes.

That kernel work is what makes the headline result possible: MoE-ViE-H, with 3.5B total parameters but only 1.1B activated, matches the zero-shot accuracy of PE-core G, a dense encoder with 1.9B activated parameters — 1.7x larger — while running at roughly 76% of its latency. On the notoriously hard ImageNet-A benchmark, MoE-ViE-H actually beats PE-core G by 0.6 points despite the parameter gap.

Teaching it video without forgetting images

The encoder is also trained for video, and that stage has its own failure mode: pretrain on images, then finetune on video-text pairs, and image performance degrades sharply — the model forgets what it learned. Mixing image data back into video finetuning helps a little but caps the video gains. The fix that works is two-part: frame-level distillation, where a frozen copy of the image-pretrained model serves as a teacher and the video-finetuned student is regularized to keep its per-frame logits close to the teacher's, plus freezing the MoE experts — and the text tower's MLP layers — during video finetuning, so only attention adapts how frame features get aggregated. Distillation and freezing each help on their own, but the combination is what preserves image accuracy while still improving video understanding.

What it adds up to

Aligned with an LLM through a three-stage pipeline (projector warmup, pre-alignment, then full supervised finetuning), MoE-ViE-H's 1.1B activated parameters outperform PE-core G, SigLIP2, and InternViT2.5 — the last with 5.5B activated parameters, five times as many — on image and video benchmarks when paired with either Llama 3.1 8B Instruct or Qwen2.5-VL 7B as the base LLM. It trails only slightly on captioning, where it lands second among sub-2B-activated-parameter models. Code is released at github.com/facebookresearch/moe_vie.

What stands out to me is that no single idea here is the whole story. Fine-grained routing, sigmoid gating, magnitude-aware balancing, and the Triton kernel are each incremental on their own — the ablations show that clearly — but together they close a gap that a decade of MoE-for-LLMs progress hadn't automatically transferred to vision. That matches what I keep seeing in production systems generally: the algorithmic idea is rarely the bottleneck, the systems engineering needed to realize its theoretical benefit is. A vision encoder underlies every VLM built on top of it, so cutting its latency at matched accuracy is a lever that compounds downstream.