Agent Lightning v1.0: RL training on the agent you actually ship

Microsoft Research Asia has released Agent Lightning v1.0, a full rebuild of their agentic reinforcement learning framework. The control plane is about 3,500 lines of code. The main idea is that the agent harness you deploy is the same one that takes part in RL training. You don't rewrite it inside a trainer.

The problem it targets

Most agentic RL stacks (the post names verl, AReaL and slime) assume the trainer owns the interaction loop. To train an agent, you rebuild its loop inside the RL framework.

Production harnesses no longer fit that assumption. Coding agents like mini-SWE-agent, OpenHands, OpenCode, Claude Code and Codex each have their own context management, tool protocols and execution logic. Reimplementing one is expensive, and the authors point out that the copy may not behave like the deployed agent. I have seen this train/serve skew in practice: a policy tuned against a simplified loop learns habits the real harness does not reward.

What v1.0 does

Agent Lightning puts an OpenAI-compatible LLM proxy between the agent and the model. You point the harness's model endpoint at the proxy, and the framework records every call. The researchers call this Harnessed Agentic RL. There are three components:

• API Gateway. The LLM proxy. It links each model call to its rollout, with the prompts, responses and log-probabilities training needs.
• Rollout Controller. It launches agents as local processes or standard Kubernetes jobs, with no dependence on commercial sandboxes.
• Customized Trainer. Built on verl, it assembles per-rollout samples into training batches.

The harnesses in the diagram are examples of what can sit behind the proxy. The reported experiments used mini-SWE-agent.

The part I find most interesting

Once the trainer only sees request/response pairs, one rollout can split into a variable number of samples. The post lists four resulting problems:

1. Retokenization. Re-rendering text context through the chat template can shift token boundaries, so adjacent calls can't always be merged.
2. Advantage calculation. Per-sample baselines count rollouts that produce more samples more than once.
3. Loss normalization. Averaging by sample count overweights rollouts that happen to fragment more, which reflects harness behavior, not task quality.
4. Backend scheduling. Sample counts are known only after the harness finishes, but GPU layout is fixed in advance.

Points 2 and 3 are a statistics problem: the unit of credit should be the episode. The authors report that rollout-level advantage and normalization gave higher validation reward and more stable policy entropy than per-sample handling. Anyone running RL behind a proxy will hit this.

On throughput, Collocated Async RL runs rollouts and updates on the same GPUs. When enough rollouts are collected, the gateway pauses new requests, drains in-flight ones, updates, and resumes, invisibly to the harness. The authors report about a 2x speedup over synchronous RL, with fewer GPUs than a conventional asynchronous setup.

The result, in proportion

The reference pipeline uses SWE-smith, mini-SWE-agent and Qwen3.5-9B with about 6,000 training samples. RL alone raised Pass@1 on SWE-bench Verified from 41.8% to 56.4%.

That is strong for a 9B model, but it is one model, one harness and one benchmark. I'd want to see whether the gain holds on harnesses with heavy context summarization or subagents, where fragmentation is worst.

My take

The value is mostly engineering. 3,500 lines is small enough to read in an afternoon and adapt. Kubernetes-job rollouts run on infrastructure you already have. And training the harness you deploy removes a whole class of train/serve mismatch. If you want to try RL on a production agent without forking it, this is one of the most direct starting points. The technical report and code are linked from the announcement.