Zhipu GLM-5.2 · SAO Asynchronous RL: Single-Rollout Surpasses GRPO, Agent RL New Paradigm Deep Dive

Zhipu GLM-5.2 · SAO Asynchronous RL: Single-Rollout Surpasses GRPO, Agent RL New Paradigm Deep Dive

1. Introduction

On July 15, 2026, Zhipu AI founder Jie Tang announced the release of SAO (Single-rollout Asynchronous Optimization), a new paradigm for large model Agentic Reinforcement Learning, jointly developed with Tsinghua University’s KEG Lab. The algorithm has been successfully deployed in the production training pipeline of GLM-5.2 (750B-A40B).

SAO’s core breakthrough: completely abandoning GRPO’s “group-wise sampling” in favor of single-rollout sampling, combined with innovative double-side token-level clipping and Critic network optimization, achieving stable training for over 1000 steps. Results: 97.3% on AIME 2025 (GRPO: 84.2%), 29.8% on SWE-Bench Verified (GRPO: 27.0%).

2. Why Asynchronous RL?

In Agent and coding tasks, rollout lengths vary dramatically. A simple code fix might need tens of tokens, while a complex multi-step Agent task requires thousands. Synchronous RL (PPO, GRPO) must wait for the slowest rollout in each group, causing severe GPU idling.

import numpy as np

class AsyncRLPipeline:
    def simulate_rollout(self, difficulty):
        times = {"simple": (0.5, 1.5), "medium": (1.5, 4.0), 
                 "complex": (4.0, 10.0), "agent": (8.0, 30.0)}
        low, high = times.get(difficulty, (0.5, 2.0))
        return np.random.uniform(low, high)
    
    def run_batch(self, tasks):
        times = [self.simulate_rollout(t) for t in tasks]
        max_time = max(times)
        total_compute = sum(times)
        sync_time = max_time * len(tasks)
        return {"gpu_util": total_compute / sync_time * 100, 
                "speedup": sync_time / max_time}

# Agent tasks: 2 agent + 3 complex + 3 medium
tasks = ["agent"] * 2 + ["complex"] * 3 + ["medium"] * 3
async_p = AsyncRLPipeline()
result = async_p.run_batch(tasks)
print(f"Async RL: GPU util={result['gpu_util']:.0f}%, Speedup={result['speedup']:.1f}x")

3. SAO’s Three Core Innovations

3.1 Innovation 1: Single-Rollout + Direct Importance Sampling

SAO replaces GRPO’s group-wise sampling with single-rollout sampling. Each prompt generates one rollout, which is immediately used for training. To address the higher variance, SAO introduces Direct Importance Sampling (DIS):

import torch
import torch.nn as nn

class DirectImportanceSampling:
    def __init__(self, clip_lower=0.2, clip_upper=5.0):
        self.clip_lower = clip_lower
        self.clip_upper = clip_upper
        self.stats = {"total": 0, "masked_low": 0, "masked_high": 0}
    
    def compute_ratio(self, current_log_probs, behavior_log_probs):
        return torch.exp(current_log_probs - behavior_log_probs)
    
    def apply_clipping(self, ratios, advantages):
        mask_low = ratios < self.clip_lower
        mask_high = ratios > self.clip_upper
        self.stats["total"] += ratios.numel()
        self.stats["masked_low"] += mask_low.sum().item()
        self.stats["masked_high"] += mask_high.sum().item()
        # Tokens outside trust region: masked out (zero gradient)
        return torch.where(mask_low | mask_high, 
                          torch.zeros_like(advantages), 
                          ratios * advantages)

Key difference from PPO: PPO clips ratio to [1-ε, 1+ε] boundaries. SAO-DIS completely masks out tokens outside the trust region, preventing any gradient contribution from extreme deviations.

3.2 Innovation 2: Critic Network Optimization

SAO re-introduces the Critic (value) network that GRPO abandoned. Three key optimizations:

package main

import "fmt"

type SAOCriticConfig struct {
    ActorUpdateSteps  int
    CriticUpdateSteps int  // 2x: Critic updates twice per Actor update
    FreezeAttention   bool // Freeze attention layers
    TrainMoEOnly      bool // Only train MoE projection layers
    SkipObservation   bool // Skip observation tokens in GAE
}

func (c *SAOCriticConfig) CalculateTrainable(totalParams, attnParams, moeParams int64) map[string]int64 {
    result := make(map[string]int64)
    if c.FreezeAttention {
        result["attention_frozen"] = attnParams
        result["attention_trainable"] = 0
    } else {
        result["attention_trainable"] = attnParams
    }
    if c.TrainMoEOnly {
        result["moe_trainable"] = moeParams
    }
    result["total_trainable"] = result["attention_trainable"] + result["moe_trainable"]
    result["frozen"] = totalParams - result["total_trainable"]
    result["freeze_ratio"] = result["frozen"] * 100 / totalParams
    return result
}

func main() {
    cfg := &SAOCriticConfig{
        CriticUpdateSteps: 2, FreezeAttention: true, TrainMoEOnly: true,
    }
    // GLM-5.2: 750B total, ~5B attention, ~550B MoE
    params := cfg.CalculateTrainable(750_000_000_000, 5_373_952_000, 549_755_813_888)
    fmt.Printf("Trainable: %d (%.1f%%)\n", params["total_trainable"], 
        float64(params["total_trainable"])*100/750_000_000_000)
    fmt.Printf("Frozen: %d (%.1f%%)\n", params["frozen"], 
        float64(params["freeze_ratio"]))
}

3.3 Innovation 3: Skip-Observation GAE

In multi-turn Agent interactions, model actions and environment observations alternate. If observation tokens participate in advantage computation, they introduce noise. SAO’s Skip-Observation GAE skips observation tokens when computing advantages:

import numpy as np

class SkipObservationGAE:
    def __init__(self, gamma=0.99, lambda_=0.95):
        self.gamma = gamma
        self.lambda_ = lambda_
    
    def compute(self, rewards, values, token_types):
        seq_len = len(rewards)
        advantages = np.zeros(seq_len)
        last_gae = 0.0
        
        for t in reversed(range(seq_len)):
            next_value = values[t+1] if t+1 < seq_len else 0.0
            
            if token_types[t] == 0:  # Action token: normal GAE
                delta = rewards[t] + self.gamma * next_value - values[t]
                last_gae = delta + self.gamma * self.lambda_ * last_gae
                advantages[t] = last_gae
            else:  # Observation token: skip (advantage = 0)
                advantages[t] = 0.0
                # last_gae unchanged, observation doesn't break GAE propagation
        
        action_mask = token_types == 0
        return advantages, {
            "snr": np.mean(advantages[action_mask]) / max(np.std(advantages[action_mask]), 1e-10),
            "skip_ratio": np.sum(~action_mask) / seq_len,
        }

# Simulate: 5 turns of action-observation alternation
seq_len = 100
token_types = np.array([0]*10 + [1]*10 + [0]*15 + [1]*10 + [0]*10 + [1]*5 + 
                       [0]*10 + [1]*10 + [0]*10 + [1]*10)
rewards = np.zeros(seq_len)
rewards[-1] = 1.0  # Final success reward
values = np.random.randn(seq_len) * 0.1 + 0.5

gae = SkipObservationGAE()
advantages, stats = gae.compute(rewards, values, token_types)
print(f"Skip-Obs GAE: SNR={stats['snr']:.2f}, Skip ratio={stats['skip_ratio']:.1%}")

4. Benchmark Results

Benchmark Base GRPO GRPO+DIS SAO
AIME 2025 72.1% 84.2% 86.5% 97.3%
SWE-Bench Verified 23.0% 25.1% 27.0% 29.8%
BeyondAIME 58.3% 65.7% 68.2% 74.8%
IMOAnswerBench 45.2% 52.8% 54.1% 61.5%

5. Conclusion

SAO provides a new paradigm for Agentic RL in large language models:

  1. Single-rollout replaces group-wise sampling: Breaks free from GRPO’s group-waiting constraint
  2. DIS with strict clipping: Tokens outside trust region are completely masked
  3. Critic network triple optimization: 2x update frequency, frozen attention, skip-observation GAE
  4. 1000-step stable training: 97.3% on AIME 2025, 29.8% on SWE-Bench

SAO has been successfully deployed in GLM-5.2 (750B-A40B) training pipeline, marking the transition of LLM RL from “synchronous batch” to “asynchronous single-rollout” era.

Technical details based on arXiv:2607.07508 and Zhipu AI official announcements.