MerchantBench and the 27% coherence gap in long-horizon AI agents

Large language model agents are good at bounded tasks with a clear finish line: book the flight, fix the bug, answer the query. What they do with a job that never ends — where today's decision constrains next month's options, and today's mistake doesn't show up in the numbers for a week — is a much harder question, and one production systems care about far more than most leaderboard benchmarks test. A new paper, MerchantBench, gives that question a concrete, uncomfortable answer.

The setup: run a store for a year, with real inventory

The MerchantBench authors gave eight production LLMs — GPT-5.6 Sol, Claude Opus 4.8, Qwen3.7-Max, Qwen3.7-Plus, GLM-5.2, DeepSeek-V4-Pro, DeepSeek-V4-Flash, and Kimi K2.6 — control of a simulated e-commerce storefront, stocked from a catalog of 98,843 real product records, and let each one run the business for 365 simulated days. Each model had 26 tools available across five categories: product sourcing (market reports, catalog search, supplier profiles), listing and pricing, cash-flow management, supplier and order monitoring, and agent-support tools like memory read/write. Every model ran under two agent scaffolds — a minimal ReAct loop and a more capable "Hermes" configuration with built-in planning, code execution, and memory management — for 48 runs in total.

The design choice that matters most is what the paper calls Mixed-Latency Feedback Adaptation. An agent's decisions generate two very different feedback streams. Supplier-side events — price changes, delistings, shipment delays — surface immediately. Order outcomes — cancellations, refunds, bad reviews — take an average of 168-plus hours, over a week, to resolve. An agent has to keep dozens of order lifecycles open simultaneously and revisit decisions it made days or weeks earlier once the delayed signal finally lands.

The headline number

After 365 days, the best configuration in the study — Qwen3.7-Max under the Hermes scaffold — closed with final net assets of roughly RMB 59,460, against a human baseline of RMB 217,610. That is 27.3% of human performance, and it was the best result in the entire study. The weakest model, DeepSeek-V4-Pro, finished at RMB 6,560 — barely above a naive rule-based baseline of RMB 24,480, and worse than doing nothing coherent at all.

Scaffold mattered more than most people would guess: switching from ReAct to Hermes lifted average net assets by 53.3%, GMV by 71.5%, and order volume by 71.2% — though not uniformly; Kimi K2.6 actually did 4.1% worse under Hermes. Model choice and agent architecture aren't separable variables in results like this, which is itself a useful data point if you're deciding where to spend engineering time.

Where the coherence actually breaks

The paper's most useful contribution isn't the topline score — it's naming the specific ways models decayed over the year:

• Activity decay. Sustained Window Rate, the fraction of decision windows in which an agent took a real operational action, dropped from 64% to 37% across the run. Agents didn't fail loudly; they just did less.
• Premature abandonment. One agent concluded on day 104 that recovery was impossible and then took no action in 355 of the remaining 523 decision windows — it never revisited that conclusion against new evidence.
• Control-loop narrowing. Supply-chain checks grew from 14% to 34% of all tool calls over the run while proactive sourcing shrank — agents drifted from strategy toward reactive firefighting.
• Flat pricing under moving costs. Human participants raised mean procurement prices from RMB 43–53 to RMB 59–91 across the year as costs shifted; GLM-5.2 and DeepSeek-V4-Flash left listing prices essentially untouched.
• Portfolio collapse from an uncorrected inference. Claude Opus 4.8's active listings shrank from 47 to 3 after the agent inferred, incorrectly, that delisting weaker products would concentrate traffic on the rest — a memory error that compounded rather than self-corrected.

None of these are single-turn reasoning failures. They're the accumulation of small, locally defensible choices into a strategy that no longer resembles one — which is exactly what "long-term coherence" is supposed to measure, and exactly what bounded-task benchmarks structurally cannot see.

This genre of benchmark is getting crowded — and that's informative

If MerchantBench's finding sounds familiar, it should. Vending-Bench, from Andon Labs, ran a similar test with a simulated vending-machine business and found the same pattern: models handle the mechanics fine but derail over long horizons, with high run-to-run variance. CoffeeBench and RetailBench apply the same premise to multi-agent coffee-shop economies and retail environments, respectively. MerchantBench's own framing places it in that lineage rather than outside it.

I don't think that overlap is a reason to dismiss the paper. What MerchantBench adds is scale and grounding: 98,843 real product records instead of a synthetic catalog, 26 tools instead of a handful, and a feedback structure — immediate supplier events against week-plus-delayed order outcomes — that maps closely onto how e-commerce operations actually run. When four independent groups build four different long-horizon business simulations and land on the same qualitative answer — models are locally competent and globally incoherent — that convergence is more informative than any single benchmark's specific number. It reads less like a novel failure mode and more like a well-replicated one, now measured with sharper instruments.

What this means if you're the one deploying the agent

I run agentic AI systems in production, and the operational lesson here isn't "current models can't be trusted with autonomy" — it's narrower and more actionable than that. The failure modes above are largely legible and, more importantly, external to the model. Sustained Window Rate is a metric you can compute and alert on. Control-loop narrowing shows up as a measurable shift in tool-call distribution. Portfolio collapse from a bad inference is exactly the kind of decision a periodic external sanity check — not a cleverer prompt — would catch before it compounds across another 250 decision windows.

The practical takeaway from MerchantBench isn't "wait for a better model." It's that any agent with a mandate longer than a single session needs state tracking and health checks that live outside the agent's own context, because the agent, by MerchantBench's own numbers, will not reliably notice its own drift.