A On-policy multi-turn RL
The environment must respond after every agent turn.
Full-trajectory policy optimization requires an interactive environment inside the training loop.
Research · Multi-turn reinforcement learning
Multi-turn RL with Step-Level Q-functions
Batch-online multi-turn RL via single-turn RL. A step-level Qπ forecasts the environment’s expected downstream return for each response, without simulating possibly random trajectories during policy improvement.
01 / The setting
Multi-turn policy improvement without a simulator in the training loop.
Many conversational agents are designed to drive outcomes such as resolving a support issue, completing a task, or helping a customer reach a decision. These interactions unfold over multiple turns, and success may only be known after the full conversation ends.
Batch-online deployment
Training proceeds in deployment rounds.
Real-world deployment is rarely fully online or fully offline. In the batch-online setting, the current policy runs over a deployment window—such as an A/B test lasting days or weeks—and logs complete conversations with observed outcomes. Training then happens offline on that fixed batch before the updated policy is deployed again.
B Iterative GRPO
Qπ forecasts the environment's expected downstream return.
Each candidate response can be scored without a simulator generating the remaining conversation.
02 / The method
Forecast the environment's expected downstream return.
At each deployment round, I-GRPO learns a step-level Q-function Qπ from logged trajectories. It forecasts the expected downstream return from the environment for each candidate response. Here, one step is one agent response conditioned on the full conversation history. I-GRPO then runs GRPO using Qπ as the reward model, without querying a simulator during policy improvement.
03 / Evaluation suite
Six multi-turn negotiation environments.
The suite spans distributive bargaining, resource allocation, and multi-issue agreement. Each environment changes what the parties know, what they can trade, and what counts as a successful outcome.
Craigslist
A vendor and customer negotiate one price. The vendor sees the listing price; only the customer sees their target.
- Private state
- Customer target
- Outcome
- Agreed price ÷ list price
Ultimatum
The proposer divides a fixed pool. The responder can accept, reject, or counter—but disagreement destroys the pool for both.
- Leverage
- Responder can destroy the pool
- Outcome
- Proposer share ÷ equal split
Deal or No Deal
Books, hats, and balls must all be allocated. Each side knows only its own per-item values, so useful trades have to be discovered in dialogue.
- Private state
- Different item values
- Closure
- Valid proposal + explicit accept
CaSiNo
Two campers divide three packages each of food, water, and firewood. Private priority rankings and reasons create opportunities for logrolling.
- Private state
- High / medium / low priorities
- Opportunity
- Trade across differing needs
Job Interview
A recruiter and worker negotiate five terms at once. Salary and holiday preferences oppose; the value of a position also depends on the company.
- Trade space
- 2 numeric + 3 discrete issues
- Dependency
- Company changes position value
Alliance
Two stakeholders negotiate five project issues. Each has a private option-score matrix and a private minimum acceptable score—their BATNA.
- Private state
- Option scores + BATNA
- Feasibility
- Both thresholds must be met
Open-source implementation
Code and experimental setup.
The release includes both model backbones, all six environments, step-level Q-function training, GRPO and baseline trainers, evaluation tooling, cached Qwen round-0 rollouts, and portable SLURM launchers.
04 / Results
Held-out evaluation.
I-GRPO achieves the highest held-out expected discounted return in five of six environments and improves over the initial policy in all six. Results use four deployment rounds, 100 in-training kernels per environment, and four conversations per kernel.
Held-out evaluation
All methods, all environments
Qwen2.5-7B-Instruct · best in-training checkpoint · 100 held-out kernels · mean ± one standard error across three seeds.
| Method | Craigslist | Ultimatum | Deal or No Deal | CaSiNo | Job Interview | Alliance |
|---|---|---|---|---|---|---|
| Base | 0.36 | 0.80 | 0.52 | 0.38 | 0.25 | 0.48 |
| MT-SFT | 0.41 ± .01 | 0.79 ± .01 | 0.50 ± .05 | 0.42 ± .01 | 0.25 ± .00 | 0.51 ± .02 |
| MT-KTO | 0.48 ± .01 | 0.86 ± .01 | 0.56 ± .00 | 0.54 ± .01 | 0.29 ± .01 | 0.59 ± .00 |
| MT-PPO | 0.38 ± .02 | 0.78 ± .01 | 0.51 ± .03 | 0.38 ± .02 | 0.24 ± .01 | 0.51 ± .05 |
| ArCHer | 0.45 ± .01 | 0.91 ± .03 | 0.60 ± .03 | 0.53 ± .03 | 0.26 ± .01 | 0.57 ± .01 |
| ArCHer-LC | 0.38 ± .01 | 0.86 ± .02 | 0.53 ± .04 | 0.40 ± .04 | 0.25 ± .01 | 0.49 ± .06 |
| ST-GRPO | 0.54 ± .00 | 0.94 ± .05 | 0.58 ± .06 | 0.46 ± .03 | 0.28 ± .04 | 0.55 ± .02 |
| I-GRPO | 0.55 ± .01 | 0.99 ± .00 | 0.70 ± .01 | 0.45 ± .03 | 0.35 ± .03 | 0.65 ± .01 |
05 / Citation
Cite Iterative GRPO.
If Iterative GRPO is useful in your research, please cite the paper.
@misc{jiang2026iterativegrpobatchonlinemultiturn,
title = {Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHF},
author = {Daniel R. Jiang and Ankur Samanta and Yukai Yang and
Jalaj Bhandari and Rémi Munos and Tyler Lu},
year = {2026},
eprint = {2511.21638},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2511.21638}
}