Why VLMs choke in the middle of long documents

Most visual document understanding benchmarks — DocVQA, ChartQA, MMLongBench-Doc — throw a real-world PDF at a model and score the answer. That tells you a model failed, not why. A dense annual report varies length, layout, modality mix, and question difficulty all at once, so when a model gets an answer wrong you cannot attribute the failure to any one factor. SynthDocBench, from Abhigya Verma and collaborators, sidesteps that confound by generating every document synthetically and varying one factor at a time.

The construction

The benchmark builds documents end to end with an LLM pipeline, using a combinatorial design across six layout archetypes. Each document is assembled from independently controlled factors: length, layout structure, modality composition (text, tables, charts), and question type. Because the documents are generated rather than scraped, the authors can hold three factors fixed and sweep the fourth — something you cannot do with a corpus of real PDFs, where length and layout complexity are entangled by default.

One detail worth calling out: a 40 percent random override is injected into the generation process specifically to stop models from exploiting spurious correlations — the same failure mode that let earlier VQA benchmarks be gamed by models that learned dataset artifacts instead of the actual visual-reasoning task. It's a deliberate design choice, not an incidental one, and it's the kind of thing you only think to add once you've watched a benchmark get solved by pattern-matching rather than reasoning.

The result is a set of long-context documents with substantially greater length and structural diversity than existing benchmarks, evaluated against seven frontier VLMs.

Three failure modes that a real-document benchmark can't isolate

The paper reports three findings, each of which depends on the factorial design to be legible:

Sharp degradation with length. Accuracy drops as documents get longer — not surprising on its own, but here it's cleanly separable from layout or content effects, so the drop can be attributed to length specifically rather than to longer documents also being more visually complex.

Positional sensitivity. This is the sharpest result. The middle third of a document is the hardest region for five of the six models tested, and five of six show a negative early-to-late accuracy trend, with the steepest decline measured at 8.3 percentage points. In other words: performance isn't just noisy across position, it's shaped like a valley, worst in the middle, recovering somewhat toward the end.

Chart comprehension breakdown in long-document settings. Charts embedded in short documents are handled reasonably; the same charts embedded deep in a long document see comprehension collapse. That's a modality-by-length interaction that a benchmark without independent factor control would just read as "charts are sometimes hard."

Taken together, the authors' framing is blunt: these results suggest current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.

Where this lands, and where it doesn't

The positional-sensitivity result will read as familiar to anyone who's tracked the text-only "lost in the middle" literature — the finding that LLMs retrieve information less reliably from the middle of a long context than from the start or end. SynthDocBench is effectively showing the same shape of failure in the visual-document setting, which is a useful confirmation but not, by itself, a new phenomenon. What's new is the attribution: because the documents are synthetic and factorially controlled, the paper can say the middle-third effect holds independent of layout and modality, rather than merely correlating with them.

That controllability is also the benchmark's limiting factor. Every document is LLM-generated, and however diverse the six layout archetypes and the 40 percent randomization make them, they're not real invoices, real contracts, or real scientific papers with their attendant noise, inconsistent formatting, and domain-specific visual conventions. A model that handles SynthDocBench's synthetic middle-third valley well is not thereby guaranteed to handle a real 80-page filing with scanned exhibits and inconsistent OCR quality. The paper's own contribution — isolating variables — is exactly what makes the benchmark less representative of the messy documents production systems actually ingest.

Why this matters if you're building on VLMs

If you're doing retrieval-augmented generation or long-document QA over visual content — contracts, financial filings, technical manuals with embedded charts — the practical takeaway isn't "avoid long documents." It's that you should not assume uniform reliability across document position. If your pipeline needs a fact from the middle third of a long visual document, don't treat a single-pass VLM read as sufficient; consider chunking with overlap, multi-pass verification, or explicit retrieval-then-verify architectures that don't rely on the model attending equally well to every page. The chart-comprehension finding is a second, separate warning: don't assume a model's chart-reading accuracy on a short isolated test transfers to charts buried on page 40 of a real report. Test position and modality-in-context specifically, not just modality in isolation.

SynthDocBench is accepted at COLM 2026 and its value is methodological as much as empirical — it's a template for how to build a benchmark that can attribute failure to cause, which is a harder and more useful problem than building one that just measures aggregate accuracy.