Google Developers Blog

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

8.5内容质量

TL;DR · AI 摘要

Google通过Zero-TIM技术消除了TPU上RL训练中的数值差异,实现真正稳定的on-policy学习。

核心要点

  • Zero-TIM技术在64个TPU v5p芯片上验证,消除训练与推理间的数值差异。
  • bfloat16精度不足导致浮点运算顺序差异,影响模型稳定性。
  • JAX的形状特化导致采样器与训练器计算路径不一致,引发数值偏差。

结构提纲

按章节快速跳转。

  1. 大语言模型的强化学习对数值稳定性高度敏感,采样器与训练器间的差异会导致训练不稳定。

  2. 浮点运算的非结合性导致不同执行路径下相同权重产生不同logprobs。

  3. ›Zero-TIM解决方案

    通过统一执行路径和精度,消除采样与训练间的数值差异,实现真正on-policy学习。

  4. 在64个TPU v5p芯片上验证Zero-TIM,实现稳定训练和推理一致性。

思维导图

用一张图看清主题之间的关系。

查看大纲文本(无障碍 / 无 JS 友好)
  • Zero-TIM技术
    • 问题背景
      • 数值不稳定性
      • on-policy与off-policy差距
    • 技术实现
      • 统一执行路径
      • 消除浮点差异
      • TPU验证

金句 / Highlights

值得收藏与分享的关键句。

  • Floating-point addition is not associative, so changing the tree changes where rounding occurs.

    — 第 2 段

    ⬇︎ 下载 PNG𝕏 分享到 X
  • The same issue appears in model reductions: matrix products accumulate partial sums, normalization computes statistics, and tensor-parallel shards combine partial outputs.

    — 第 2 段

    ⬇︎ 下载 PNG𝕏 分享到 X
  • Zero-TIM implementation closes this numerical gap, assigning bitwise-identical logprobs to the same action and token history.

    — 第 1 段

    ⬇︎ 下载 PNG𝕏 分享到 X
#TPU#强化学习#数值稳定性#Google#JAX
打开原文

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch - Google Developers Blog

Google Tag Manager (noscript)

End Google Tag Manager (noscript)

HTML

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

NOV. 16, 2026

Yuxuan Zhang

Software Engineer

Lin Chai

Senior Software Engineer

Lance Wang

Tianshu Bao

Senior Staff Software Engineer

Weiren Yu

Senior Product Manager

Share

  • Facebook
  • Twitter
  • LinkedIn
  • Mail

The problem

Reinforcement learning (RL) for large language models is notoriously sensitive to numerical instability: even tiny discrepancies between rollout generation and gradient updates can destabilize training or stall convergence. In LLM RL, the model serves as the policy, π(action | previous tokens), and true on-policy learning requires every action in an update to be generated and evaluated by that exact same policy. While a synchronous loop ensures each rollout batch is scored using the same weights that generated it, a critical numerical requirement remains: the sampler and trainer must evaluate that policy identically.

In practice, production RL stacks run sampling and training through separate, specialized execution paths. The sampler chooses each action token during incremental decoding and records its behavior logprob. The trainer evaluates the same token in batched forward-backward passes to compute its current logprob and gradients. Under the same weights, these differing execution paths can still assign different logprobs to the same token and context. The training schedule is on-policy, but the numerical execution has introduced a hidden off-policy gap that compounds over multi-turn trajectories.

Our zero train–inference mismatch (Zero-TIM) implementation, developed in Tunix and verified at multi-host scale across 64 TPU v5p chips (DP8×TP8), closes this numerical gap. At the same weights, the sampler and trainer assign bitwise-identical logprobs to the same action and token history—bringing state-of-the-art, true on-policy stability to TPU RL training.

Why it happens

A reduction is a tree of floating-point additions. Floating-point addition is not associative, so changing the tree changes where rounding occurs. The effect is easier to expose in bfloat16 , which retains less precision than float32.

The following bfloat16 example isolates the effect of reduction order:

Figure 1. Near 2.0, each 0.0078125 input is half the distance to the next representable bfloat16 value. Round-to-nearest-even discards each input when added separately; grouping the same inputs first reaches the next value.

The same issue appears in model reductions: matrix products accumulate partial sums, normalization computes statistics, and tensor-parallel shards combine partial outputs. In our original stack, sampler decode and trainer forward used different token-array shapes and forward implementations. JAX specializes computations to their input shapes, and the training forward also runs inside a differentiated program.

Across these paths, reduction order and intermediate precision were not consistently aligned. Floating-point rounding turned those arithmetic differences into different logprob bits. The weights and model equations were identical; the arithmetic was not.

The alignment contract

The contract is simple: across sampler decode, serving prefill, and trainer forward, all three paths must produce bitwise-identical logprobs for every sampled action—comparing raw floating-point bytes under the same rollout weights, token history, and sampling settings, with zero rounding or tolerance.

Why three paths instead of two? In a decoupled RL stack, a direct sampler–trainer disagreement shows that a numerical gap exists, but not where it enters. Introducing serving prefill as a middle checkpoint turns alignment into a systematic two-step diagnostic that any RL stack can adopt :

Sampler decode vs. prefill verifies that the serving engine agrees with itself between incremental token generation and full-sequence re-scoring, while prefill vs. trainer verifies that the serving and training engines agree across compilation and differentiation boundaries.

Figure 2. Bitwise comparisons of sampled-action logprobs within the serving engine and across the serving–training boundary, evaluated at the same rollout weights.

How we built zero-TIM on TPU

Achieving Zero-TIM does not require merging the sampler and trainer into a single monolithic engine. Instead, we use a generalizable four-part recipe that bridges separate serving and training pipelines across multi-host TPU meshes: locking within-shard kernel accumulation, ordering cross-shard reductions, standardizing the logprob scoring tail, and wrapping serving kernels with custom differentiation rules.

Fix within-shard arithmetic

Inside each tensor-parallel rank, batch-invariant Pallas kernels control accumulation order and intermediate precision. Projection kernels traverse fixed contraction blocks in order, keeping the reduction blocking independent of token-row count. Partial products accumulate in float32, with explicit output casts.

The production kernel initializes its scratch accumulator before the tile traversal, then applies this core to each tile:

code
input_tile = x_ref[...]
weight_tile = y_ref[...]

partial = jnp.dot(
    input_tile,
    weight_tile,
    preferred_element_type=jnp.float32,
)
acc_ref[...] = acc_ref[...] + partial
out_ref[...] = acc_ref[...].astype(out_ref.dtype)

Python

Copied

The next tile continues from the float32 accumulator, not the cast output. This construction covers the attention and MLP projections and the LM head. Dedicated kernels fix RMSNorm reductions and SwiGLU’s intermediate casts. The existing Pallas paged-attention forward is retained.

Order cross-shard reductions

For row-parallel projections, each tensor-parallel rank produces a partial output. We separate communication from addition: ppermute transports the partials, then every rank adds them in the same source-rank order.

The reduction order is explicit in the program rather than selected by a collective implementation.

Share the action-logprob path

Alignment must extend beyond model logits to the probability of the sampled action. Decode, prefill, and trainer apply matching sampling transforms, use the same fixed Pallas log-softmax construction, and select the recorded action token.

Prefill and trainer score the action generated during rollout rather than choosing another action. We also preserve sampled token IDs across environment turns so that every path evaluates the same token history.

Differentiate the serving forward

The trainer reuses the serving prefill computation. Custom vector–Jacobian products retain the serving kernels for forward evaluation and provide their backward rules. The matmul example shows this separation:

code
@jax.custom_vjp
def serving_matmul(inputs, weights):
    return pallas_matmul(inputs, weights)

def matmul_forward(inputs, weights):
    output = pallas_matmul(inputs, weights)
    return output, (inputs, weights)

def matmul_backward(saved, output_gradient):
    inputs, weights = saved
    _, pullback = jax.vjp(
        differentiable_matmul,
        inputs,
        weights,
    )
    return pullback(output_gradient)

serving_matmul.defvjp(matmul_forward, matmul_backward)

The differentiable replica mirrors the kernel’s accumulation and casts, but is used only to construct the pullback. Forward values still come from the serving Pallas kernel.

Training places that forward inside a differentiated JAX program. We pass --xla_allow_excess_precision=false through XLA_FLAGS to the compiler process, constraining excess intermediate precision in the operations surrounding the Pallas kernels.

Long trajectories are evaluated in fixed-size chunks, carrying the paged KV cache forward. All action logprobs feed the complete train-batch RL loss. The reverse pass propagates gradients through those cache connections and combines parameter-gradient contributions before the optimizer update.

Figure 3. KV-cache state connects the forward chunks; cache gradients connect their reverse passes. Parameter-gradient contributions are combined before the optimizer update.

Bringing frontier RL numerical alignment to TPU

Across the AI systems community, eliminating train–inference mismatch has rapidly become a state-of-the-art requirement for stabilizing large-scale RL convergence. Recent open-source efforts—including vLLM with TorchTitan and RadixArk/LMSys Miles —have demonstrated how batch-invariant kernels and bitwise numerical alignment prevent off-policy drift during policy optimization.

Our work delivers an open-source, modular JAX/XLA and Pallas implementation of this contract for TPU in Tunix. Verified across a multi-host 64 TPU v5p (DP8×TP8) mesh , these kernel and collective patterns provide a reusable blueprint that teams can adopt to achieve bit-exact on-policy RL across distributed JAX/TPU stacks.

Evaluation and results

To benchmark our TPU kernels, we run a 200-step numerical verification test on the synthetic FrozenLake environment using a public Qwen3-8B checkpoint as a baseline test case using 64 TPU v5p chips with a DP8×TP8 mesh. The recipe uses a GSPO-token loss with leave-one-out advantages. We compare Standard training with trainer-old logprobs and no additional sampler correction, token-level truncated importance sampling (TIS), and Zero-TIM.

The Zero-TIM configuration also uses token-in/token-out (TiTO) handling to preserve sampled token IDs across environment turns. The comparison runs retain their existing tokenization paths.

Figure 4. Single-seed training runs on FrozenLake grids from 2×2 through 9×9, with up to five interaction turns. Both panels show the first 200 training steps. Logprob differences are plotted without smoothing; solve rates show faint raw measurements and a ten-step trailing average. Endpoint labels show the final ten-step averages.

Zero-TIM records zero mean absolute sampler–trainer logprob difference at every logged step in this window, while its training solve rate increases. Standard and TIS retain nonzero differences: TIS adjusts their contribution to the update without removing the underlying mismatch. Zero-TIM reaches higher observed solve rates in these runs.

Code and updates

We developed Zero-TIM prototype as part of Tunix , Google's open-source, JAX-native library for LLM RL and post-training on Cloud TPU. The kernel patterns and three-way verification contract are modular by design—making them straightforward to adopt across existing multi-host JAX/TPU serving and training pipelines—and we are actively extending support to linear-attention and Mixture-of-Experts (MoE) architectures.

Preview the current implementation

posted in:

  • AI
  • Cloud
  • Tutorials
  • Case Studies
  • How-To Guides
  • Announcements
  • Learn
  • Solve
  • Tunix
  • Reinforcement Learning
  • Pallas
  • Cloud TPU

Previous

Next

Related Posts

List

Mobile

Web

Tutorials

How-To Guides

A2UI 1.0: Build delightful, fast, reliable, agentic experiences

NOV. 11, 2026

AI

Cloud

Best Practices

Problem-Solving

Agents are sessions, not requests: size GKE node pools for AI agent sessions

Securing Your Agents: A Hands-On Guide to Agent Anomaly Detection

Navigation dots