Apple Machine Learning Research

Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps

8.5内容质量

TL;DR · AI 摘要

苹果提出在线强化学习框架,通过Planning-Grounding代理提升iOS应用UI导航性能,实验显示其泛化能力超SFT基线7倍。

核心要点

  • GiGPPO方法通过视觉相似状态奖励信号提升critic稳定性
  • 8B RL代理在Android环境泛化能力达SFT基线7倍
  • 基础设施支持数百虚拟iOS设备并发训练

结构提纲

按章节快速跳转。

  1. 指出当前UI导航模型在实时应用中的局限性。

  2. 提出Planning agent与UI Grounding模型的双组件架构。

  3. LLM-as-a-Judge提供密集步级反馈与终端奖励结合。

  4. 采用PPO与GiGPPO方法实现双奖励结构优化。

  5. 基于视觉相似状态的邻居条件价值估计提升稳定性。

  6. 在UINavBench等基准测试中显著超越SFT和终端RL基线。

思维导图

用一张图看清主题之间的关系。

查看大纲文本(无障碍 / 无 JS 友好)
  • 在线强化学习UI导航框架
    • 核心架构
      • Planning agent
      • UI Grounding模型
    • 关键技术
      • LLM-as-a-Judge奖励机制
      • GiGPPO价值估计
    • 实验成果
      • UINavBench基准测试
      • Android环境泛化能力

金句 / Highlights

值得收藏与分享的关键句。

#强化学习#UI导航#Apple#iOS#PPO
打开原文

Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps - Apple Machine Learning Research

research area

Human-Computer Interaction

,

Methods and Algorithms

content type

paper

published

November 2026

Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps

Authors Peitong Duan*, Seyed Khashaiar Gatmiry*, Jeff Nichols, Emmanuel Abbe, Samy Bengio

View publication

Copy Bibtex

While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.

  • * Equal contribution

Related readings and updates.

ILuvUI: Instruction-Tuned Language-Vision Modeling of UIs from Machine Conversations

July 14, 2025 research area Human-Computer Interaction conference IUI

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations,…

Read more

From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts

June 30, 2025 research area Human-Computer Interaction conference IUI

With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actions—particularly those that may be risky or irreversible—remain under-explored. In this work, we investigate the real-world impacts and…

Discover opportunities in Machine Learning.

Our research in machine learning breaks new ground every day.

Work with us