🌊 Tide

Information moves like the tide. We pick the waves worth studying.

中文

← Today's tide

Published 2026-07-08 · Z.ai · Zhenyu Hou, Yujiang Li, Jie Tang et al.

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

arXiv · HuggingFace · ▲15

Abstract (verbatim)

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

Background

Background Analysis

1. Technical Context
Recent advances in large language models (LLMs) have shifted focus from supervised pretraining to reinforcement learning (RL) for post-training improvement. The core goal is to enhance model "intelligence" for complex tasks like coding or mathematical reasoning. However, traditional RL pipelines suffer from inefficiency: most approaches use synchronous batch processing, where the policy generates a batch of trajectories before optimization. This is problematic for "long-horizon" tasks (e.g., multi-step coding), as varying trajectory completion times lead to GPU underutilization.

2. Previous Limitations
While asynchronous RL improves resource efficiency, it introduces two critical issues:
- Training instability: Trajectories may be generated by different versions of the old policy, exacerbating "off-policy" effects and causing training volatility.
- Mismatched sampling: Methods like GRPO rely on "group-wise sampling," where multiple responses per prompt are averaged. This conflicts with asynchronous environments—groups must wait for the slowest trajectory, wasting resources. Group sampling also fails in online scenarios with dynamic feedback.

3. Proposed Solution
The paper introduces Single-rollout Asynchronous Optimization (SAO), addressing these issues with three key designs:
- Single-rollout sampling: Each prompt uses only one trajectory for updates, reducing off-policy effects and avoiding group delays.
- Strict token-level clipping: Directly uses probabilities from the trajectory generator with tighter clipping to stabilize training.
- Improved value model training: Updates the critic more frequently and freezes attention layers to reduce noise.

4. Key Differences
Compared to prior work, SAO’s innovations lie in:
- Balancing efficiency and stability: Unlike previous asynchronous RL, which prioritized throughput, SAO ensures both efficiency and training stability via single-rollout sampling and clipping.
- Adaptability to online settings: SAO’s sampling and advantage estimation (GAE) are designed for dynamic environments, whereas methods like GRPO excel in static batch tasks.

SAO outperforms GRPO and its variants in coding/math benchmarks (e.g., SWE-Bench) and was deployed for training the 750B-parameter GLM-5.2 model.

Method, Figure by Figure

M E T H O D1 / 3

Figure 2: Overview of SAO with single rollout design. The numbers denote the generation order of trajectories. For SAO, each trajectory becomes available for training immediately upon completion. In contrast, GRPO must wait until all trajectories in a group are generated before training can begin.

This diagram compares the workflows of two reinforcement learning (RL) methods: GRPO (the traditional method at the top) and SAO (the method proposed in this paper at the bottom). The core difference lies in the timing and manner of trajectory generation and training.

GRPO Section (Upper Workflow):

  1. Rollout Module: Multiple "Rollout" (trajectory generation) modules work in parallel to generate different trajectories. The numbers (e.g., 4, 6, 8; 1, 2, 7, 9; 3, 5) indicate the generation order of the trajectories—trajectories in different groups need to be generated within their groups before moving to the next step.
  2. Waiting for Group: GRPO adopts a group - wise sampling strategy. That is, it must wait until all trajectories in a group (such as 4, 6, 8 in the first group; 1, 2, 7, 9 in the second group; 3, 5 in the third group? Or more accurately, multiple trajectories within a group) are generated before collecting these trajectories and passing them to the "Training" module. This means that trajectories cannot be used for training individually; all trajectories in a group must be generated first.
  3. Training Module: After receiving a complete set of trajectories, it trains using the policy \(\pi_{\theta}\).
  4. Trust Region Diagram on the Right: It shows the constraints on policy updates. The vertical axis is the probability ratio of the policy \(\frac{\pi_{\theta}}{\pi_{\text{rollout}}}\) (or a similar form), and the horizontal axis is the action \(A\). There are two dashed lines (maybe \(1+\epsilon_{l}\) and \(1 - \epsilon_{l}\)? Or \(1+\epsilon_{h}\) and \(1 - \epsilon_{l}\)? It needs to be combined with the context, but the core is to limit the range of policy updates to ensure stability. However, the group sampling in GRPO leads to late training timing, while SAO changes this.

SAO Section (Lower Workflow, Method in This Paper):

  1. Rollout Module: There are also "Rollout" modules to generate trajectories, but the numbers (such as 9, 8, …, 3, 2, 1) here indicate that a single trajectory can be used for training immediately after generation ("Single - Rollout Ready for training"). That is, after each trajectory (corresponding to a prompt) is generated, it does not need to wait for other trajectories and directly enters the "Training" module.
  2. Training Module: After receiving a single trajectory (rather than a group), it trains using the policy \(\pi_{\theta}\). This method reduces the "off - policy" effect because the trajectory is used for training immediately after generation, which is closer to the online learning scenario.
  3. Improved Training Design: SAO reduces the off - policy effect and improves generalization ability through "single - rollout sampling" (one trajectory corresponds to one prompt), and at the same time, combines practical value model training design (not detailed in the diagram, but mentions "practical value - model training designs").
  4. Optimization Stability: SAO introduces a strict bilateral token - level clipping strategy (not directly shown in the diagram, but part of the method) to improve optimization stability.

Core Operation Mode of the Methods (Inferred from the Diagram):

  • Problem of GRPO: Group sampling leads to the situation that trajectories can only be trained after all trajectories in a group are generated, which reduces efficiency and may increase the off - policy effect (because the generation times of trajectories are different, and the policy may have changed when training).
  • Solution of SAO:
  • Replace group sampling with single - trajectory sampling: After a trajectory (corresponding to a prompt) is generated, it is immediately used for training ("Single - Rollout Ready for training"). In this way, the generation and training of trajectories are almost synchronous, reducing the waiting time and also reducing the off - policy effect (because the policy during training is closer to the policy when the trajectory is generated).
  • Combine practical training design (such as value model training) and strict clipping strategy to improve training stability and task performance.

Comparison and Conclusion (Inferred from the Diagram and Abstract):

  • Efficiency and Stability: The single - trajectory strategy of SAO makes training can be carried out more timely, while the group strategy of GRPO needs to wait for all trajectories in a group to be generated. SAO performs better in training stability (such as being able to train stably for a thousand steps) and task performance (better than GRPO and its variants in agentic coding and reasoning benchmark tests).
  • Applicable Scenarios: SAO is particularly effective in simulating online learning scenarios, because the model needs to adapt to changing environments (or tasks), and the immediate training of single trajectories is more in line with the needs of online learning.

In summary, this diagram clearly shows how SAO solves the problems of stability and off - policy effect in asynchronous RL by changing the timing and manner of trajectory generation and training, thus surpassing traditional methods in terms of efficiency and effectiveness, through the comparison between GRPO's "group - waiting - batch training" and SAO's "single - trajectory - immediate training".

(a) We report the accuracy transition of three writing styles—cute, chuunibyou, and classical—on a held-out evaluation set throughout the online training process. Shaded regions indicate phase transitions where the reward preference is switched to favor a different stylistic archetype. SAO rapidly suppresses the previously dominant style and realigns its policy to the new target based on environmental feedback. (b) We compare the evolution of training rewards between SAO and a Running Mean Advantage Estimation baseline under single-rollout online learning. Shaded regions denote stylistic reward shifts. While both methods eventually recover after distribution changes, the Running Mean baseline exhibits a pronounced adaptation lag and lower stable performance. Figure 5: Online learning simulation under changing writing-style preferences.

This figure (Figure 5a) illustrates the accuracy changes of models for three different writing styles ("Cute," "Chuunibyou," and "Classical") during the online training process, verifying the online learning capability of the proposed Single-rollout Asynchronous Optimization (SAO) method when style preferences dynamically change.

Components of the Figure and Information Flow

  • X-axis (Training Step): Represents the training steps, ranging from 0 to 400, showing the progression of the training process over time.
  • Y-axis (Accuracy (%)): Represents the model's accuracy on a held-out evaluation set, measuring the model's ability to generate content that matches the corresponding style.
  • Four curves: Correspond to three styles: "Cute" (cyan), "Chuunibyou" (purple), "Classical" (orange), and "Academic" (blue). "Academic" might be a baseline or another category, but the focus is mainly on the three target styles.
  • Shaded regions: Indicate stages where style preferences switch, i.e., periods when the reward preference shifts from one style to another. For example, the shaded regions around 150-200 steps and 300-350 steps are critical periods for style switching.

How the Method Works (Revealed from the Figure)

The core of the SAO method is to quickly adjust the strategy based on environmental feedback to adapt to new style preferences: 1. Initial Stage: Before the style switch (non-shaded regions), the model has a higher accuracy for a certain style (e.g., "Cute" in the early stage, "Chuunibyou" in the middle stage), indicating that the current strategy of the model is more inclined to generate content of that style. 2. During Style Switch (Shaded Regions): When the reward preference switches (the start of the shaded region), the model needs to quickly adjust to adapt to the new target style. For example, in the first shaded region (after about 150 steps), the accuracy of the previously dominant "Cute" style drops rapidly (the cyan curve drops from about 60% to near 0), while the accuracy of the "Chuunibyou" style (the purple curve) rises rapidly (from near 0 to about 75%). This shows that SAO can quickly suppress the previously dominant style and re-adjust the strategy according to environmental feedback (new reward preference) to align with the new target style. 3. Subsequent Style Switch: In the second shaded region (after about 300 steps), the accuracy of the "Chuunibyou" style drops rapidly (the purple curve drops from about 75% to near 0), while the accuracy of the "Classical" style (the orange curve) rises rapidly (from near 0 to about 60%), further verifying the fast adaptation ability of SAO.

Interpretation of the Results (Coordinates, Comparison, and Conclusion)

  • Coordinates and Data Points: The training steps on the x-axis range from 0 to 400, and the accuracy on the y-axis ranges from 0 to 75%. For example, at about 100 training steps, the accuracy of the "Cute" style reaches about 60%, making it the dominant style at that time; after the shaded region around 200 steps, the accuracy of the "Chuunibyou" style reaches about 75%, becoming the new dominant style; after the shaded region around 350 steps, the accuracy of the "Classical" style reaches about 60%.
  • Comparison Objects: Mainly the accuracy change curves of the three target styles ("Cute," "Chuunibyou," and "Classical"), as well as the change of the "Academic" curve (but "Academic" may be a control and not the main focus).
  • Conclusion: SAO can quickly suppress the previous dominant style and quickly align with the new target style when style preferences change dynamically (through reward switching in the shaded regions), demonstrating its effectiveness and stability in the online learning scenario (learning scenario with changing style preferences). Compared with the Running Mean Advantage Estimation baseline mentioned in the paper (described in Figure 5b), SAO has a faster adaptation speed and higher stability (although Figure 5a mainly shows the adaptation process of SAO itself, its advantages can be known from the caption).

In short, this figure proves that the SAO method can quickly adjust its strategy to adapt to new reward preferences in the online learning scenario (with dynamically changing style preferences) by showing the accuracy changes of three writing styles at different training steps and the fast adaptation process during style preference switching, thus performing excellently in style generation tasks.

Figure 6: Training reward for token-level SAO training and step-level variants, where token-level shows better training rewards.

This figure shows the change of training reward during the reinforcement learning training process for different methods, helping us understand the advantage of the Single - rollout Asynchronous Optimization (SAO) method over step - level variants.

First, look at the axes: The horizontal axis is "Training Step", with a range from 0 to 400, representing the progress of training; the vertical axis is "Reward", with a range from 0.42 to 0.54. Generally, a higher reward indicates better performance of the model during training.

Then, look at the three curves, which represent different methods: - The curve with blue triangles is the "SAO" method. From the figure, we can see that as the training steps increase, the reward of SAO shows an overall upward trend. In the later stage of training (for example, near 400 steps), the reward value is obviously higher than the other two methods, reaching close to 0.54. - The curve with purple diamonds is the "Step - level(Average)" method. Its reward also increases with the increase of training steps, but the speed of increase and the final reward value are both lower than those of SAO. During the training process, its reward fluctuates relatively greatly. For example, there is an obvious peak around 200 steps, then it decreases a bit and continues to rise, but the overall reward is lower than that of SAO. - The curve with blue circles is the "Step - level(Last - Token)" method. The reward of this curve also increases with the increase of training steps, but similar to "Step - level(Average)", its reward value and upward trend are not as good as those of SAO, and the reward value is obviously lower than that of SAO in the later stage of training.

From the results of this figure, we can draw the following conclusion: In token - level SAO training, its training reward is higher than that of the two step - level variants (average and last - token step - level methods). This shows that the SAO method can obtain better training rewards during the training process, which means that SAO may have better performance in reinforcement learning training. This also verifies the conclusion in the paper that SAO can be stably trained and perform well in related benchmark tests.

From the perspective of how the method works, SAO uses single - rollout sampling (one rollout per prompt) to reduce off - policy effects and improve generalization, and at the same time introduces a strict bilateral token - level clipping strategy to improve optimization stability. From this figure of training rewards, we can see that the growth trend of the reward of SAO during training is better, which shows that these designs make SAO learn more effectively during training, thus obtaining a higher reward.

← → flip · ↑ ↓ scroll notes · Esc exit