Don't Fix 'How Many Steps to Think': HKU-Tsinghua VAST Lets Agents Adapt Their Time Horizon by Value, 2.5x the Best Baseline on Hard Tasks
Don’t Fix “How Many Steps to Think”: HKU & Tsinghua’s VAST Lets Agents Adapt Their Time Horizon by Value, Scoring 2.5x the Best Baseline on Hard Tasks
Part One: The Root of Long-Horizon RL Is in “Value”
There is an old saying in reinforcement learning: everything is value. An agent knows which way to move because its value function estimates how much future return a present choice buys. Estimate value accurately and the policy learns well. But as tasks grow long and staged, this estimate degrades.
In June 2026, Kexin Zheng (HKU), Xianyuan Zhan (Tsinghua AIR), and Xintao Yan (HKU) posted “Horizon Adaptive Offline Policy Learning via Value Stitching” (VAST) on arXiv (2606.21136); it was accepted to NeurIPS 2026, with code open on GitHub (Whiterrrrr/value-stitching). It targets a fundamental but often bypassed problem in offline RL: when updating value, how far into the future should you look?
Start with the disease. Classic one-step temporal-difference (TD) learning updates current value using “immediate reward + a discounted estimate of the next state’s value”:
V(s) ← r(s,a) + γ · V(s')
In theory, repeatedly applying the Bellman operator converges to the optimal value function — but only in an ideal world of unlimited samples and zero function-approximation error. In reality data is finite and the network’s own V(s') is already inaccurate; worse, that inaccurate estimate is used as the learning target to teach the previous state, so error propagates back along the decision chain and compounds. This is the accumulation of bootstrapping bias — the longer the horizon, the further the chain, the bigger the snowball.
A natural fix is to “look a few more steps”: n-step TD includes the real rewards of the next n steps before bootstrapping, and methods like Q-chunking model fixed-length action segments. The more real reward you use, the less you rely on “estimating yourself,” and the lighter the bias in principle.
But extending the horizon is not the same as choosing the right horizon. That is VAST’s entry point.
Part Two: The Fixed-Step Dilemma — Why “Look More Steps” Treats the Symptom
Real long-horizon tasks have sharply different phases. Picture directing a robot arm to arrange three colored blocks into a pattern: reach, grasp, transport, align, release — each sub-phase has a completely different rhythm. Yet n-step TD and Q-chunking pin down one span n (say, fixed at 5) and cut every state and every phase with the same rigid ruler.
That ruler fails at both ends:
- Span too short: the key turning point lies beyond view, so value must still “guess” the far future through repeated bootstrapping and short-range bias accumulates;
- Span too long: a critical move (the final alignment) gets scored in the same chunk as irrelevant transition steps, blurring the short-term decision that actually matters.
The deeper contradiction: a task’s key transitions never land exactly on the boundaries you pre-drew. Measuring an uneven-rhythm task with an equidistant ruler systematically misaligns the cuts. The paper puts it bluntly: existing methods use a “rigid, fixed-horizon recipe,” too inflexible to capture the complex value structure of long-horizon, multi-stage tasks.
Fixed n-step (one size for all) VAST adaptive (pick the anchor by value)
s0──s1──s2──s3──s4──s5 s0 s3 s6
|──── 5 steps ───| |── G(s,s3,3) ─|
|──── 5 steps ───| |── G(s3,s6,3) + γ³V(s6)
boundaries misaligned ✗ anchor pinned at the real sub-goal ✓
one ruler for all states coarse far / fine near, state-dependent
This matches how humans decide. The paper offers an analogy: when planning a distant goal, a person forms only a coarse sub-goal with a rough horizon (“walk straight about a hundred meters, then turn right”); for an immediate critical decision, they focus exclusively on near-term, actionable consequences (“slow down, steer slightly left to avoid the obstacle”). Coarse far, fine near, varying with state — a mixed-horizon modeling need.
The classic option / semi-Markov framework and hierarchical RL have touched this idea, but the former is structurally heavy and the latter expresses only two fixed scales (high and low). A scheme that freely and adaptively chooses any horizon between 1 and K and is practical to deploy was largely missing. VAST fills that gap.
Part Three: The G-Function — First Learn to Price “Any Segment”
To freely stitch futures of different lengths, you must answer a concrete question: starting from state s and arriving at a future state s_k after k steps, how much is that segment worth?
VAST introduces an auxiliary value function, the G-function (horizon-based cumulative return):
G(s, s_k, k) = expected discounted cumulative return of going from s to s_k over k steps
Note the difference from an ordinary value function: plain V(s) takes one start and looks all the way to episode end; G takes three inputs — start s, future anchor s_k, horizon k — and measures the return between two reachable states over a specified number of steps. It is a tape measure you can set to any length from 1 to K.
The G-function trains directly with offline-data supervision: draw any segment from an offline trajectory, and the discounted sum of the real rewards along that segment is a natural label — no bootstrapping, no reliance on the network’s own estimates. At the source, this avoids the “teaching yourself with your own inaccurate output” loop.
But pricing isolated fragments is not enough; the model might just memorize segments seen in training. VAST adds a compositional consistency constraint: a long return should equal “the G of the front piece + the discounted G/value of the back piece” when cut at a midpoint. This forces coherence across time scales so the model generalizes to (s, s_k, k) combinations never directly seen.
class GFunction:
"""Auxiliary value: discounted return of going from s to s_k over k steps.
Supervised directly by real segment returns from offline data — no bootstrap."""
def __init__(self, gamma=0.99):
self.gamma = gamma
self.table = {}
def label_from_segment(self, rewards):
g = 0.0 # a real segment's discounted return
for t, r in enumerate(rewards):
g += (self.gamma ** t) * r
return g
def fit(self, s, s_k, k, g):
self.table[(s, s_k, k)] = g # direct supervision
def __call__(self, s, s_k, k):
return self.table.get((s, s_k, k), 0.0)
def consistency_loss(self, s, s_m, s_k, m, k):
whole = self(s, s_k, k) # whole ≈ front m + γ^m · back
split = self(s, s_m, m) + (self.gamma ** m) * self(s_m, s_k, k - m)
return (whole - split) ** 2
Direct-return supervision supplies the “data grounding”; the composition constraint supplies “cross-scale consistency.” Together they make G a tape whose markings are accurate and that can reach positions it never saw.
Part Four: The Stitching Bellman Operator — Free the Bootstrap Anchor from “Next State” to “Any Sub-Goal”
With G in hand, VAST rewrites the value update itself. The ordinary Bellman equation anchors the bootstrap at “the next state s′,” horizon always 1. Since G can price any segment, we can release that anchor:
(stitching Bellman operator)
(T_stitch V)(s) = max_{k ∈ {1..K}, s_k} G(s, s_k, k) + γ^k · V(s_k)
Read plainly: across every horizon k and every reachable future state s_k, choose the combination maximizing “this segment’s return G + the discounted value beyond the anchor γ^k·V(s_k).” The chosen s_k is the most worthwhile intermediate sub-goal; k is the scale at which to view it. When k=1, G(s,s₁,1) collapses to the immediate reward r and the formula recovers the standard Bellman equation — so one-step TD is just a special case of VAST.
The elegance is that it unifies “how far” and “where to” into one maximization: the bootstrap anchor is no longer passively whatever the environment hands you next, but actively placed at the sub-goal with the most sensible value structure. It strides across meaningless transitions far away, and bootstraps in fine steps at critical nodes.
Crucially, this is not an ad-hoc heuristic — the paper proves the stitching Bellman operator is a γ-contraction: repeatedly applied to any bounded value function, it converges to a unique fixed point V*. Intuitively, whatever k the inner max chooses, its discount γ^k is at most γ (k≥1), so the gap between value functions across iterations shrinks at rate at least γ. Contraction answers the question reviewers most want asked — does freely choosing horizons destabilize or fail to converge? — with a clean “no.”
def stitch_backup(G, V, transitions, s, K=8, gamma=0.99):
"""VAST stitching backup: pick the best (k, s_k) over horizons and reachable states."""
best = float("-inf")
best_choice = None
for k in range(1, K + 1):
for s_k, reachable in transitions.reachable(s, k):
if not reachable:
continue
candidate = G(s, s_k, k) + (gamma ** k) * V(s_k)
if candidate > best:
best, best_choice = candidate, (k, s_k)
return best, best_choice # update target + best (horizon, sub-goal)
def nstep_backup(rewards, V, s_n, n, gamma=0.99):
"""Control: fixed n-step, one ruler; anchor passively lands at step n."""
g = sum((gamma ** t) * r for t, r in enumerate(rewards[:n]))
return g + (gamma ** n) * V(s_n) # cannot pick anchor or vary n by state
Side by side the difference is stark: fixed n-step passively accepts “wherever step n lands,” while stitching actively searches a table of “horizon × future state” for the optimum. This is the mechanistic core of why VAST’s value learning is more accurate and far more stable on long-horizon tasks.
Part Five: Two-Level Policy — the Brain That Picks Sub-Goals, the Hand That Walks
Accurate value must still become action. VAST cleanly splits the policy into two layers with distinct duties:
The upper layer is the stitching policy π_stitch. It reads the current state and outputs “which future sub-goal s_k, at what horizon k.” It is trained advantage-weighted: higher-value “state–horizon” combinations in the data get greater weight, biasing the policy toward anchors proven to lead to high return; a diffusion/flow model enhances expressiveness over the sub-goal distribution.
The lower layer is the execution policy π_exec. Given “current state + designated sub-goal,” it produces the concrete primitive action, learned with goal-conditioned behavior cloning straight from offline trajectories. At test time rejection sampling is layered on: it draws several candidate actions toward the chosen sub-goal, scores them with a Q-function, and executes the highest-value one.
current state s
|
v
+---------------------+
| π_stitch policy | pick (k, s_k): where to go + how far (adv-weight/diffusion)
+---------------------+
| sub-goal s_k (note: k is NOT passed down)
v
+---------------------+
| π_exec policy | goal-conditioned BC emits primitive actions
+---------------------+
| several candidate actions
v
Q rejection-sampling, take the best ──► execute
A detail worth savoring: the execution policy deliberately does not receive the horizon k. Why? Training data comes from an existing, possibly sub-optimal behavior policy whose implicit horizons reflect the “old rhythm”; at inference the stitching policy gives an optimized “new rhythm.” If π_exec depended on k during training, there would be a train-test mismatch between the sub-optimal training k and the optimized inference k. So k is simply withheld — the hand only needs to know the current sub-goal and how to reach it; how many steps away it is, is none of its business. The brain owns the rhythm; the hand walks the current segment. Split the duties and the mismatch vanishes.
The loop is complete: first use value to choose a future worth heading to, then use value to filter the actionable move at hand. The same sub-goal information serves both value supervision and action generation, joining “how to evaluate the future” and “how to move toward it.”
Writing the execution policy out in a few lines makes the two-level split and the rejection-sampling step concrete. Note that during training pi_exec receives only “current state + sub-goal” and never sees k; k lives entirely in the upper pi_stitch decision:
class ExecPolicy:
"""Lower execution policy: goal-conditioned BC for actions, Q rejection sampling.
Key: horizon k is absent from the inputs — the hand only needs the sub-goal."""
def __init__(self, q_net):
self.q = q_net
def bc_loss(self, s, s_k, a_expert):
a_pred = self.policy(s, s_k) # given (s, sub-goal), not k
return ((a_pred - a_expert) ** 2).mean() # behavior cloning from offline data
def act(self, s, s_k, n_candidates=8):
cands = [self.policy.sample(s, s_k) for _ in range(n_candidates)]
return max(cands, key=lambda a: self.q(s, a, s_k)) # Q scores candidates, take best
There is an engineering judgment worth lingering on: more information is not always better. Intuitively, telling the execution layer “how many steps remain” (k) seems to help it plan; but training data comes from a possibly sub-optimal behavior policy whose implicit horizons differ from the upper layer’s optimized new rhythm. If pi_exec depended on k during training, deployment would feed it a different k distribution and force extrapolation on an input condition it never saw. VAST simply withholds k — trading a little interface-level information for train-test consistency. It is a textbook move: kill a distribution mismatch with a structural constraint, and a pattern worth borrowing when building agent systems.
Part Six: 50 OGBench Tasks — The Edge Is on Multi-Stage Hard Tasks
Experiments run on OGBench, a goal-conditioned offline RL benchmark of challenging robotic locomotion and manipulation: whole-body humanoid control, maze navigation, block manipulation. Configuration:
- 50 state-based tasks across 10 domains: a Basic set of 30 tasks across 6 common domains and a Hard set of 20 tasks across 4 high-planning-complexity domains;
- 8 baselines spanning the main routes: one-step value (IQL, FQL, IFQL), fixed 5-step (FQL-n, IFQL-n), chunked value (QC), and generative value (VALUE-FLOWS, FLOQ);
- each method trained offline for 1 million steps; the main table uses 8 random seeds, 50 episodes per task — a solid protocol.
VAST is best or tied for best in both domain averages and the overall score, with gains most pronounced on scene, cube, and puzzle — the multi-stage manipulation tasks requiring complex planning, exactly where a fixed span is most easily mis-cut. The most telling number: VAST is the strongest method across every Hard domain, with an overall score 2.5x the best baseline. Its learning curve keeps climbing in the Hard setting, evidence that the horizon-aware paradigm genuinely scales with complexity rather than only winning on easy tasks.
A telling visualization on cube-triple-play-task3 overlays the ground-truth trajectory with VAST’s predicted future states: the chosen horizons are not mechanically equidistant but track the task’s staged “atomic intentions” — striding when it should, stopping when it should, producing a clearly phased path of intermediate targets that progressively translates the distant final goal into immediately actionable moves. Adaptive horizons change not just “how far to look” but “where the next segment should lead.”
Part Seven: Lessons and Limits — It Is Offline RL, Not a Ready-Made Agent Harness
The boundaries, and the bridge to today’s agent time-planning.
First, VAST is an offline RL method, not an online agent harness. All training relies on a pre-collected static dataset with no environment interaction; value stitching and the two-level policy are proven and evaluated in the offline paradigm. Equating it directly to “letting a ChatGPT-style agent decide adaptively how many steps to execute” is overreach — the online, high-dimensional, language-driven, strongly stochastic setting poses different challenges.
Second, experiments use state inputs, not raw pixels. The 50 tasks use structured states; moving to vision/touch and real sensing still needs representation learning. The conclusion openly notes the VAST core can combine with diffusion policies, model-based methods, and representation learning — an open direction, not a finished item.
Third, the core insight transfers to agent design. This week we discussed “commitment depth”: a long-horizon system plans an action sequence, executes it open-loop, then re-observes — how long should that sequence be? Too short means constant replanning and extra forward passes; too long means imagined and realized states diverge and errors compound. A separate study (arXiv:2605.09860) independently reaches nearly the same conclusion from the online VLM side — when the optimal depth varies across states, learning depth as a function of state strictly dominates any fixed scalar. From the offline value-learning end, VAST gives the same philosophy a rigorous form via the G-function and a provably convergent stitching operator: time horizon should not be a hand-tuned hyperparameter but a control variable decided dynamically by state and value.
The practical nudge for agent builders: if you still hard-code “plan N steps / replan every N steps,” then no matter how N is tuned it is suboptimal on uneven real tasks. The better investment is letting the model judge whether the current situation deserves a longer single commitment or an immediate pause to re-observe the world — handing “when to recommit” back to value itself.
Sources: Zheng, K., Zhan, X., Yan, X., “Horizon Adaptive Offline Policy Learning via Value Stitching,” NeurIPS 2026, arXiv:2606.21136, code at github.com/Whiterrrrr/value-stitching; Tsinghua AIR, The University of Hong Kong; Sina Finance / AIR commentary. The G-function, stitching Bellman operator, OGBench 50 tasks, and 2.5x Hard score are from the paper; Python is a simplified illustrative re-implementation, not the official code.