Animated I-GRPO diagram: collect multi-turn episodes, fit Q-pi on Monte Carlo returns, score a group of responses with Q-pi, run GRPO, and repeat with the improved policy.
I-GRPO
0.5×
1×
2×
Restart
Iteration
1
Policy success rate
30%
Future turns simulated while scoring
0
Numbers are illustrative.