How two API settings tripled a reasoning model's benchmark score
OpenAI tripled GPT-5.6 Sol's ARC-AGI-3 score by turning on two existing Responses API settings — retained reasoning and context compaction — showing that harness design, not just model capability, decides what actually reaches an agent's task performance.
OpenAI published a short piece this week that says more about how we operate reasoning models than it does about model capability: How enabling two settings tripled our scores on the ARC-AGI-3 benchmark. The finding is blunt. GPT-5.6 Sol — a model that, per OpenAI's own account, has already been used to solve open problems in mathematics — scored 13.3% on the ARC-AGI-3 public task set under the official evaluation harness. Running the same model through OpenAI's Responses API with two settings turned on, retained reasoning and compaction, pushed that score to 38.3%, using roughly six times fewer output tokens per game. No new model, no new training run. Same weights, different plumbing.
What actually changed
ARC-AGI-3 is a suite of 2D puzzle-style games the model has to play across many turns, inferring rules it's never told directly. Each move depends on what it learned from the moves before. That makes it a state-holding task, and it turns out the official harness was fighting the model's own architecture rather than exercising it.
Two things were off:
- Reasoning was discarded after every move. GPT-5.6 Sol, like other reasoning models, does private chain-of-thought before it acts. The Responses API lets that reasoning persist across turns when you chain calls with
previous_response_id. The official harness wasn't doing that — every move was closer to a fresh, stateless call. The model had no memory of its own prior thinking, only the visible transcript of what it did. - Older history was truncated, not summarized. Once the running context grew past a threshold, the harness dropped the oldest turns outright. Compaction is a Responses API feature that instead folds older turns into a compact carry-forward item — opaque, not meant to be human-readable, but preserving the state needed to keep playing rather than deleting it.
Put together, the model was re-deriving each game's rules from scratch on nearly every move, then losing that work anyway once the transcript got long. Retaining reasoning and compacting context let it keep its own reasoning and play history instead, so it stopped re-solving problems it had already solved. That's the whole story behind both the score jump and the token drop.
OpenAI's before/after: reasoning discarded and history truncated each turn, versus retained reasoning and context compaction carried forward.
Why this is the actual news
It would be easy to read this as an ARC-AGI-3 leaderboard story. I'd read it as a harness-design story, and that's the more useful lesson. As reasoning models get deployed as agents — multi-turn, multi-tool, running for minutes or hours — the harness around the model — how state gets passed, what gets kept, what gets thrown away between calls — increasingly determines how much of the model's capability actually reaches the task. A benchmark score, or a production agent's success rate, is a joint measurement of the model and the plumbing it runs through. Most evaluation setups, including serious ones, are still built on API contracts from before reasoning persistence and compaction existed: stateless calls, rolling-window truncation, one shot per turn. That's a reasonable default for a single-shot chat completion. It's a quiet tax on anything that has to reason across turns.
What I'd take from it
If you're running agents against reasoning models and something is underperforming, check two things before concluding the model is the bottleneck. First, are you chaining calls so reasoning persists, or is every turn effectively wiping the model's scratch work? Second, when context grows past your window, are you truncating or compacting? The failure mode here doesn't announce itself as an error — it looks like a mediocre agent. It shows up as worse plans, repeated exploration, and higher token spend for the same task, all of which read as a capability ceiling when the real constraint is state management. Before spending effort on prompting or fine-tuning to close a performance gap, it's worth first confirming the harness is actually giving the model access to its own memory.