VisionWeave: teaching a multimodal model where detail is worth paying for
VisionWeave trains an MLLM to choose, block by block, between four fine visual tokens and one coarse one. On Qwen3.8-27B it drops 43% of visual tokens at 98.9% of native quality and more than doubles SGLang throughput on a long-video workload. Here is how it works, and where I would and would not trust it.
Every multimodal model I have put into production has the same cost structure. The text prompt is cheap and the image is expensive. A native-resolution vision encoder turns a screenshot, an invoice or a video frame into a dense grid of patch tokens, and the LLM pays attention and feed-forward compute on every one of them. That holds whether a patch covers a line of 8-point text or a stretch of empty white margin.
VisionWeave (Feng et al., arXiv:2610.07987), from the Qwen team at Alibaba with collaborators at USTC, CUHK and Zhejiang University, goes straight at that mismatch. It does not prune tokens after the fact. It trains the model to decide, region by region, how much resolution each part of the image deserves. The headline result on Qwen3.8-27B: 43.0% fewer visual tokens on average across eight benchmarks, 98.9% of native performance retained, and a 2.30x end-to-end throughput gain when served on SGLang.
I think this is one of the more deployable efficiency papers I have read this year, and the reasons are mostly in the engineering details, so that is where I will spend the time.
Why pruning keeps disappointing in practice
The standard way to cut visual tokens is pruning or merging. FastV drops tokens the attention pattern says are unimportant, and VisionZip merges redundant ones. Both usually run at a preset ratio, typically 50%.
The paper's evaluation shows where that breaks. Averaged over eight benchmarks on Qwen3.8-27B, FastV and VisionZip keep only about 88% of native performance at 50% savings. The average hides the real problem. On ScreenSpotV2, a GUI grounding benchmark, VisionZip loses 47.9% of its score. On DocVQA it loses 17.8%. On video, where frames are redundant, both methods look fine.
It is easy to see why. A pruned token leaves a hole in the image. If the hole lands on the button the agent is supposed to click, no amount of downstream reasoning brings the button back. A fixed ratio also cannot tell a sparse slide from a dense spreadsheet. And attention-score pruning needs the attention weights, which FlashAttention-style kernels do not expose. That is one reason, as the authors note, pruning methods have rarely shipped in SGLang or vLLM.
The mechanism: change resolution instead of deleting
VisionWeave never deletes a region. Every 4x4 block of ViT patches can be represented in one of two ways:
- Fine: the model's native 2x2 merger produces four tokens, exactly as the base model would.
- Coarse: a gated spatial pooler first compresses each 2x2 neighborhood of patch features, then passes the result through the same native merger to produce one token in the same embedding space.
The pooler is a learned generalization of mean pooling. It layer-normalizes the four patch features, builds a shared context vector, and computes channel-wise softmax gates over the four positions. Zero-initializing its final layers makes it start out as plain mean pooling, which gives training a stable starting point. In the ablation it beats a pixel-unshuffle projector on all eight benchmarks.
A granularity router makes the choice. One learnable query per block cross-attends to ViT features at three depths (layers 0, L/2 and L), first locally to the block's own patches and then globally to the whole frame. It outputs a probability of going coarse, and the default threshold is 0.5.
The detail I like most is positional. Each coarse token sits at the geometric center of the four fine tokens it replaces, at half-integer MRoPE coordinates, so the LLM sees one coherent spatial frame whatever the mix.
Trained by self-distillation, no labels required
The training recipe is three stages, and the teacher in every stage is the model's own native forward pass. The loss is forward KL over the teacher's top-512 tokens.
- Coarse-only. Train just the pooler with every block forced coarse, so the student sees 25% of the tokens.
- Soft mixing. Train just the router, using a differentiable blend of fine and coarse features plus an MoE-style balance loss.
- Hard routing. Train the pooler and LLM on the exact discrete sequences used at inference.
The third stage matters more than it might look. It removes the train-inference mismatch that heuristic pruning always carries, and in the appendix it lifts ScreenSpotV2 from 66.6 to 91.4. The 27B run used roughly 0.78M samples and over 30K A100 GPU-hours. That is a real cost, but for anyone serving a VLM at volume it is a one-time cost.
What the numbers say
The savings track content. On Qwen3.8-27B, VisionWeave saves only 16.5% on InfoVQA and 27.3% on DocVQA, where the images are dense, and 49.6–54.2% on the three video benchmarks, where losses stay at or below 0.30%. On ScreenSpotV2 it saves 55.1% and loses 3.2%, so interfaces turn out to be mostly empty space around a few small targets.
Two controls convinced me the router is actually learning something:
- Shuffled routing. Keep the same number of coarse blocks per image but scramble their positions, and the 27B model's average drops 5.36 points, with ScreenSpotV2 down 32 points. Where you spend tokens matters, not just how many you spend.
- Matched budgets. Give FastV and VisionZip VisionWeave's own per-dataset savings ratios, and they still average 70.1 and 70.3 against VisionWeave's 76.7.
The threshold also works as a runtime dial. Dropping it from 0.5 to 0.25 on Video-MME raises savings from 54.2% to 66.0% for a 0.37-point loss. Setting it to 1 recovers the full fine-grained model. One checkpoint can serve several cost tiers.
The serving result, read carefully
The team integrated VisionWeave into SGLang. Half-integer RoPE positions are handled by doubling coordinates and halving rotary frequencies, which reuses the existing kernels. Layout metadata is attached before scheduling, so dynamic batching, chunked prefill and CUDA-graph decoding all keep working. Throughput rose from 0.90 to 2.07 requests per minute, mean TTFT fell 54.4% and mean TPOT fell 60.6%.
The benchmark setup deserves attention. It ran on two A100-80GB GPUs, with LongVideoBench at up to 256 frames and 1024 tokens per frame, at a client concurrency of four. That is a long-video workload, where VisionWeave's savings are at their highest. A document-heavy pipeline saving 16–27% of tokens should expect a smaller gain. I would benchmark on my own traffic mix before promising anyone 2x.
My take
The framing I take from this paper is that representational granularity becomes a learned degree of freedom rather than a preprocessing constant. The same idea runs through mixture-of-experts and adaptive computation, and here it is applied to the input side of the model, where multimodal costs actually concentrate.
The limitations are the ones the authors themselves state. There are only two granularity levels. The router design was not explored much, because training compute capped it. And the capability is added in post-training rather than pre-training. The SGLang code is promised at github.com/FFY0/VisionWeave but was not something I could evaluate.
If you run document agents, GUI agents or video understanding at scale, this is the first adaptive-token approach I would actually prototype against my own serving stack. It degrades gracefully on the inputs that break pruning, and it was built to fit the serving engine rather than to fight it.