Mosaic and the multi-session gap in agentic RL

Every production agent I have worked on eventually runs out of context. When it does, the harness either compacts the history into a summary and carries on, or hands part of the task to a subagent with a fresh window. Either way, the agent that finishes the task is no longer working from the context it started with. Most agentic RL pipelines quietly assume the opposite: one rollout, one continuously growing context, one training segment.

Mosaic: Multi-Session Agentic Reinforcement Learning (Xu et al., LARK at HKUST(GZ) with Huawei's Foundation Model Lab) names this train/inference mismatch and proposes a way to train across it. I think the framing is right and overdue. I am less convinced by the evidence, and I'll explain why.

The mismatch

The paper defines a session as a stretch of agent-environment interaction over a single context prefix. A trajectory becomes a set of sessions. Context compaction ends one session and starts another from a compressed prefix. Subagent forking creates branch sessions whose results are merged back into the main agent's context.

Standard GRPO treats a rollout as one linear sequence and assigns it one advantage, computed from the reward relative to the other rollouts in its group. That works when the rollout really is one sequence. Once compaction rewrites the prefix partway through, the tokens after the boundary were conditioned on a summary the model itself wrote. They were not conditioned on the history GRPO assumes. A model trained only on single-session rollouts has never been rewarded or penalised for how it summarises or delegates, yet at inference those are some of its most consequential actions.

Broadcasting, and why it is coarse

The common workaround, which the authors call broadcasting, makes multi-session rollouts trainable by copying the trajectory-level advantage to every token of every session. Each session becomes its own training segment, and the loss is token-weighted per trajectory so long rollouts do not dominate the group.

This is reasonable plumbing, but it means a compaction that threw away the one fact needed to answer the question gets exactly the same credit as the search session that found it. It also says nothing about how many sessions an agent should spend.

What Mosaic changes

Mosaic stays critic-free and adds two layers on top of GRPO.

Trajectory level: session-aware advantage. Within each group, trajectories are split by outcome. Session counts are converted to a centred rank in [-1, 1] inside each outcome subgroup, and the GRPO advantage is multiplied by (1 - rho * rank), with rho = 0.2. Successful trajectories that used fewer sessions get more credit. Failed trajectories that used more sessions are penalised less, on the logic that giving up early is the worse failure. Because the multiplier stays positive, a failure can never outrank a success. For all-success or all-failure groups, where GRPO's advantage is zero, small fallback terms still produce a session-count signal. The appendix proves the ordering properties cleanly.

Session level: marginal information gain. Each session is treated as a state transition. Its reward is the change in the policy's average log-likelihood of the ground-truth answer, measured from the context before the session to the context after it. A compaction that preserves the right facts raises that likelihood, and one that drops them lowers it. These gains are normalised separately per session type, because compaction sessions systematically score lower than search sessions and would otherwise be penalised just for being compactions. The final session is zeroed out so that the answer itself does not leak into the signal. The shaped term is added with lambda = 0.1.

The design is careful. Using ranks instead of raw counts, multiplying instead of adding, and normalising per type are the right choices, and each one is justified in the paper.

What the numbers show

The experiments train Qwen3.5-4B on 1,000 deep-search tasks for one epoch (batch 16, group size 8) and evaluate on BrowseComp.

In the compaction setting, Mosaic gains 3.5 points over broadcasting, the paper's 10.1% relative improvement, while using about a quarter fewer sessions. Session quality also improves: the share of sessions with positive information gain rises from 47.7% to 59.9%. The method's marginal-gain computation is also much cheaper than IGPO's turn-level version.

Why I read the evidence as thin

The idea deserves more than this experimental base supports.

• One benchmark, one 4B model, one short run. One epoch over 1,000 tasks at batch size 16 is about 60 policy updates by my arithmetic. No seeds or confidence intervals are reported.
• The subagent gain is small, and its uncertainty is unreported. On BrowseComp's 1,266 questions, Mosaic's 1.0-point edge over broadcasting is about 13 questions, and its edge over plain single-session GRPO is about 7. Without paired per-question outcomes or repeated runs, there is no way to tell how much of that is signal. In this setting, broadcasting actually scores below the single-session model.
• The multi-session training baseline barely helps. Under compaction, broadcasting beats single-session GRPO by 0.6 points. Almost all of the reported gain comes from Mosaic's shaping, not from training on multi-session rollouts in general. That is interesting, but it is one comparison.
• The session reward needs a reference answer. Measuring the likelihood of a known answer suits BrowseComp, whose answers are short and verifiable. It does not obviously transfer to coding or open-ended agent work, which is where compaction and subagents are used most.
• The authors' own ablation flags the DAG problem. Subagent trajectories branch, so one session's immediate gain may not reflect its downstream effect. The combined method helps much less there.

What I take from it

The lasting contribution is the problem statement. If your harness compacts or forks at inference, your RL loop should see those boundaries during training, and credit should be able to tell a good summary from a bad one. The paper's discussion makes the same point about training inside opaque harnesses, and also raises asynchronous rollouts, where multi-session variance makes group comparisons harder.

The trajectory-level session-rank modulation is cheap and principled, and I would try it first. I would treat the session-level shaping as promising but unproven until it is tested on a second task family and a larger model, with error bars.