Tsinghua + Tencent GPS Deep Dive: Guided Posterior Sampling Cuts LLM Post-Training Cost by 69% with Just 100 Samples

Tsinghua + Tencent GPS Deep Dive: Guided Posterior Sampling Cuts LLM Post-Training Cost by 69%

1. Introduction: The Post-Training Cost Crisis

The ceiling of LLM capabilities is no longer determined by pre-training — post-training (SFT, RLHF, GRPO) is what transforms a generic foundation model into a practical, usable tool. However, post-training costs are spiraling out of control.

GRPO (Group Relative Policy Optimization) requires models to generate large batches of candidate responses at each iteration, evaluated by a reward model. This Rollout process accounts for over 70% of total training costs. On July 12, 2026, Tsinghua University and Tencent published a breakthrough method — GPS (Guided Posterior Sampling) — that uses a small model to “guide” a large model’s RL post-training, reducing Rollout costs by up to 69% with just 100 cross-domain training samples.

2. The GRPO Cost Problem

2.1 Traditional GRPO Cost Structure

Traditional GRPO (70B model):
  Rollout batch: 64 candidates/step
  Training steps: 1000
  Output tokens: 512 avg/candidate
  Total cost: ~$65,536 (70%+ on Rollout)

GPS Method:
  Small model (7B) explores first: 64 candidates at 1/1000th cost
  Reward model filters: top-16 candidates
  Large model (70B) refines only in high-value regions
  Total cost: ~$20,316 (69% savings)

3. How GPS Works

3.1 Core Intuition

GPS’s intuition is elegant: use a small model to explore the answer space cheaply, then guide the large model to sample only in high-probability regions.

Traditional GRPO:
  Prompt → Large model generates 64 candidates → Reward scores → Policy update
              ↑ Extremely expensive ↑

GPS Method:
  Prompt → Small model explores (64 cheap candidates) → Filter top-K → Large model refines
              ↑ 1/1000th cost ↑                      ↑ Guided ↑

3.2 Technical Details

The small model (7B parameters) explores the answer space at 1/1000th the cost of the large model (70B). A reward model scores the candidates, selecting the top-K. The large model then performs guided sampling only in these high-value regions, with KL divergence constraints to prevent drifting from its original capabilities.

"""
GPS Sampling Simulation
"""
import numpy as np

experiments = [
    ("AIME 2025 (Math)", 72.3, 71.8, 5000, 100, 0.69),
    ("MATH-500", 89.5, 89.1, 3000, 100, 0.68),
    ("Web Search", 63.2, 67.8, 2000, 100, 0.71),
    ("GSM8K", 95.1, 95.0, 2000, 100, 0.67),
]

print(f"{'Task':25s} {'GRPO Baseline':14s} {'GPS Result':12s} {'Diff':8s} {'Samples':10s}")
print("-" * 69)
for name, grpo, gps, trad_s, gps_s, saving in experiments:
    diff = gps - grpo
    print(f"{name:25s} {grpo:>6.1f}%{'':>8s} {gps:>6.1f}%{'':>6s} {diff:>+5.1f}%  {gps_s}/{trad_s}")

print("\n\nKey Findings:")
print("  1. Only 0.5% accuracy loss on AIME while cutting costs by 69%")
print("  2. Web Search actually improved by 4.6% (guided exploration more effective)")
print("  3. Sample count reduced from thousands to 100 (95%+ reduction)")
print("  4. Only 100 cross-domain samples needed for training")

4. Why GPS Works: Theory

Intuition: Imagine finding an exit in a massive maze. Traditional GRPO sends one explorer (the large model) to randomly walk 64 paths to completion. GPS sends a scout (the small model) to quickly explore and mark promising areas, then the explorer only searches those regions carefully.

Theoretical guarantee: The small and large models’ probability distributions over high-quality answers are correlated. Regions the small model considers “good” will likely contain even better solutions for the large model. KL divergence constraints ensure the large model doesn’t drift too far from its original capabilities.

5. Implementation: GPS Sampling Engine in Go

package main

import (
	"fmt"
	"math"
	"math/rand"
	"sort"
	"time"
)

type Candidate struct {
	Score  float64
	Model  string
	Tokens int
	Cost   float64
}

type GuidedSampler struct {
	smallCost float64
	largeCost float64
	topK      int
	exploreN  int
}

func NewGuidedSampler() *GuidedSampler {
	return &GuidedSampler{
		smallCost: 0.000002,
		largeCost: 0.00002,
		topK:      16,
		exploreN:  64,
	}
}

func (gs *GuidedSampler) GuidedSample(prompt string) (float64, float64) {
	// Phase 1: Small model exploration
	candidates := make([]Candidate, gs.exploreN)
	for i := range candidates {
		tokens := 100 + rand.Intn(400)
		candidates[i] = Candidate{
			Score:  0.3 + rand.Float64()*0.5,
			Tokens: tokens,
			Cost:   float64(tokens) * gs.smallCost,
		}
	}
	
	// Phase 2: Filter top-K
	sort.Slice(candidates, func(i, j int) bool {
		return candidates[i].Score > candidates[j].Score
	})
	topCandidates := candidates[:gs.topK]
	
	// Phase 3: Large model refinement
	bestScore := 0.0
	for _, c := range topCandidates {
		if c.Score > bestScore { bestScore = c.Score }
	}
	
	tokens := 200 + rand.Intn(300)
	finalScore := math.Min(bestScore*0.7+0.3*(0.5+rand.Float64()*0.5), 1.0)
	
	totalCost := 0.0
	for _, c := range candidates { totalCost += c.Cost }
	totalCost += float64(tokens) * gs.largeCost
	
	traditionalCost := 64.0 * 400.0 * gs.largeCost
	savings := (1 - totalCost/traditionalCost) * 100
	
	fmt.Printf("GPS Score: %.4f, Cost: $%.4f, Savings: %.1f%%\n",
		finalScore, totalCost, savings)
	return finalScore, totalCost
}

func main() {
	rand.Seed(time.Now().UnixNano())
	sampler := NewGuidedSampler()
	sampler.GuidedSample("Solve math problem: A train leaves station A...")
}

6. Industry Impact: Democratizing Post-Training

For large enterprises: $45K+ savings per training run on 70B models; $650K+ for 1000B models.

For SMEs and research institutions: GPS reduces post-training requirements from “thousands of GPU hours” to “hundreds of GPU hours.” Teams with limited compute can now participate in LLM post-training optimization.

For academia: GPS opens a new paradigm of “model guiding model” — the small model is no longer just a distillation teacher, but a “scout” in reinforcement learning.

7. Conclusion

GPS (Guided Posterior Sampling) from Tsinghua and Tencent achieves 69% Rollout cost reduction and 95%+ sample reduction with less than 1% accuracy loss. This is not an incremental optimization — it’s a paradigm-level innovation proving the feasibility of “small-large model collaboration” in LLM training. As models like GPT-5.6 and DeepSeek V4 consume hundreds of millions in training costs, cost-reduction technologies like GPS may be more practically significant than performance improvements themselves.

• ←Previous
• Next→