Learn2Play Bench: do LLM agents actually learn from experience?
A new benchmark of text games with novel, counterintuitive rules separates real learning from recalled pretraining. The practical results for agent builders: raw experience logs are competitive with, and often better than, distilled rules, and the harness moves scores as much as the model does.
Most claims that an agent learns from experience have a confound baked in. If the task is familiar, such as fixing a bug, booking a flight, or playing a well-known text adventure, improvement across attempts could come from new knowledge or from the model slowly remembering what it already knew. Learn2Play Bench, from Yibo Li, Bryan Hooi and colleagues at the National University of Singapore, is built to remove that confound. I think it is one of the more useful agent evaluations of the year. The leaderboard is not the reason. The reason is that its findings translate directly into design decisions for anyone building agents with memory.
The setup: rules no model has seen
The benchmark has 20 hand-designed text games whose hidden rules are novel and sometimes deliberately counterintuitive. In one game, stealing a colleague's lunch gets you promoted and working hard does not. In Poisoner, you serve dishes to a king and have to infer his taste constraints from what he accepts and rejects. The agent receives a goal and a list of valid actions, never the rules. Feedback is deterministic and scoring is automatic, so a trajectory can be replayed and inspected.
Each agent plays repeated episodes (five for normal games, ten for the five harder challenge games) over three independent trials, with no weight updates. What carries over between episodes depends entirely on the harness. Half the games are re-shuffled: characters, menus, or layouts change each episode while the underlying rule stays fixed. That tests whether the agent learned the rule or just memorized the instance.
The paper reports four metrics on normalized scores: Max (peak), Mean (consistency), Learning Gain (last episodes minus first), and Learning Slope (a linear fit across episodes). Separating performance from improvement turns out to matter. Claude Opus 5.5 has the highest Max (80.1) and Mean (61.0) under the common OpenCode harness, but Qwen 3.8 Max has the steepest learning slope (+6.30 versus +5.22). The model that scores best is not necessarily the one that learns fastest.
Finding one: keep the evidence, not just the conclusion
This is the result I would put in front of any team building a self-improving agent. The authors compare six ways of carrying experience across episodes, holding the backbone fixed: Naive (nothing carried over), Memory (complete raw interaction logs appended to the prompt), Reflexion, EvoTest, ReasoningBank, and Agent Workflow Memory.
With Kimi K3, plain Memory leads on Max, Learning Gain, and Slope, and ties EvoTest on Mean. Max is 63.3 against 56.1 for AWM, and Learning Gain is +16.9 against +4.8. The ranking is not uniform. With Opus 5, EvoTest and Reflexion reach a higher Max than Memory. Averaged over three backbones, Reflexion edges ahead on Max while Memory still leads on Mean, Learning Gain, and Slope. So raw logs do not win everywhere, and the data do not show that more elaborate distillation reliably does worse. What they do show is that the simplest option, keeping everything, is competitive across backbones and strongest on the learning metrics.
The mechanism is the interesting part. In GemForge, where agents combine ingredients into gems, one EvoTest run tried a single fusion, got slag, and its evolved prompt then told the agent never to attempt tier-2 or tier-3 fusion. That ruled out combinations it had never tested, including ones that work. Summarization turned an incomplete belief into a standing instruction, and the original evidence that could have overturned it was gone. Raw logs keep the observation available, so a later contradiction can still revise the rule.
I read this as an epistemics problem rather than a memory-size problem. A distilled rule is a conclusion with its error bars stripped off. The authors suggest linking each learned rule to the interactions that produced it, and I agree. In production I would treat any learned rule as a hypothesis with provenance: store the claim, the evidence count, and pointers to the raw traces, and let the agent re-open it when feedback disagrees.
Finding two: the harness is half the system
With the backbone held fixed, swapping the harness changes results dramatically. Opus 5 goes from 74.5 Max in OpenCode to 81.8 in Claude Code. GPT-5.6-SOL goes from 57.0 in OpenCode to 74.3 in Codex. That second jump is bigger than the gap between several backbones on the leaderboard. The authors attribute this to models being trained alongside their native harnesses, and to harnesses being tuned to their models.
The cost result is the part teams tend to miss. A better harness was often cheaper as well as better. With Kimi K3, OpenCode used 56% fewer input tokens than the Memory method and beat it on Max, because it executes several actions per model response and does not re-send repeated game feedback. Price per token did not predict cost per episode either. Kimi K3 beat Claude Sonnet 5 at lower total cost despite higher token prices, because it used fewer tokens (0.56M versus 0.73M per episode). If you are choosing a model by comparing price sheets while holding a generic loop fixed, you are probably optimizing the wrong variable.
The flexibility point matters too. OpenCode let the model decide when to review earlier feedback, run a calculation, or check a plan. The self-evolving methods followed fixed update steps. When something unexpected happens, an agent that can stop and investigate within the episode does better than one that can only write a note for next time.
Where agents still fall short
Top human players, college students with no AI tools, beat every agent on peak score: Human Top-1 reaches 84.3 Max. The behavioral data explains why. In fixed games, humans' consecutive action sequences are less similar to each other than agents' (0.54 versus 0.66), and after a bad episode humans set a new personal best 33% of the time, against 22% for agents. Agents find something that partly works and keep exploiting it. Agents won where success required long, precise execution or computation, as in RedEye and Model Auditor.
Two other failure modes are worth naming. Agents can know a rule and still fail to plan with it. In Haunted Inn, Opus 5 correctly stated the room taboos, then assigned rooms greedily and left the last guest with no safe option. Transfer is also fragile. When surface details change while rules stay fixed, learning slopes drop, and weaker backbones drop most: Claude Sonnet 5 falls from 4.90 to 2.23. Some agents remembered which menu position worked rather than why it worked.
What I would take from this
The benchmark has limits. It uses one seed per game, three trials, and text-only games, and the human pool is students. Still, the conclusions line up with what I see in deployed systems:
- Store traces, not just lessons. If you distill experience, keep the provenance and let the agent revisit it.
- Evaluate the harness as a first-class variable. Model swaps inside a fixed generic loop understate what a model can do and can overstate its cost.
- Measure learning separately from performance. Max and Learning Slope rank systems differently, and you need both.
- Budget for exploration. Agents under-explore once something works. That is a policy choice you can design for, not a fixed property of the model.
The paper and project site include per-game curves and annotated rule-discovery evidence. Those logs are worth reading for anyone designing agent memory.