Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps
TL;DR · AI 摘要
苹果提出在线强化学习框架,通过Planning-Grounding代理提升iOS应用UI导航性能,实验显示其泛化能力超SFT基线7倍。
核心要点
- GiGPPO方法通过视觉相似状态奖励信号提升critic稳定性
- 8B RL代理在Android环境泛化能力达SFT基线7倍
- 基础设施支持数百虚拟iOS设备并发训练
结构提纲
按章节快速跳转。
思维导图
用一张图看清主题之间的关系。
查看大纲文本(无障碍 / 无 JS 友好)
- 在线强化学习UI导航框架
- 核心架构
- Planning agent
- UI Grounding模型
- 关键技术
- LLM-as-a-Judge奖励机制
- GiGPPO价值估计
- 实验成果
- UINavBench基准测试
- Android环境泛化能力
金句 / Highlights
值得收藏与分享的关键句。
GiGPPO通过跨轨迹视觉相似状态奖励信号提升critic稳定性
8B RL代理在Android环境泛化能力达SFT基线7倍
基础设施支持数百虚拟iOS设备并发训练实现在线学习
Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps - Apple Machine Learning Research
research area
Human-Computer Interaction
,
Methods and Algorithms
content type
paper
published
November 2026
Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps
Authors Peitong Duan*, Seyed Khashaiar Gatmiry*, Jeff Nichols, Emmanuel Abbe, Samy Bengio
View publication
Copy Bibtex
While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.
- * Equal contribution
Related readings and updates.
ILuvUI: Instruction-Tuned Language-Vision Modeling of UIs from Machine Conversations
July 14, 2025 research area Human-Computer Interaction conference IUI
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations,â¦
Read more
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts
June 30, 2025 research area Human-Computer Interaction conference IUI
With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actionsâparticularly those that may be risky or irreversibleâremain under-explored. In this work, we investigate the real-world impacts andâ¦
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.
Work with us