UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning
arXiv · HuggingFace · ▲68
Abstract (verbatim)
Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction. However, building multi-platform GUI agents remains challenging. On one hand, high-quality and executable cross-platform interaction trajectories are still scarce, and existing data often suffer from limited platform coverage. On the other hand, different platforms exhibit distinct interaction conventions, making joint or continual training prone to behavioral pattern mixing, platform-specific capability degradation, and catastrophic forgetting. To address these challenges, we construct Uni-GUI, a high-quality cross-platform GUI interaction dataset, and propose UI-MOPD, the first method that incorporates multi-teacher on-policy distillation into continual learning for GUI agents. UI-MOPD dynamically selects a platform-specific teacher according to the current environment and transfers platform-specific behavioral priors to a shared policy through platform-conditioned distillation, enabling adaptation to new platforms while preserving capabilities on existing ones. Experiments on OSWorld and MobileWorld show that UI-MOPD achieves task success rates of 38.2% and 12.0%, respectively, demonstrating its effectiveness in balancing cross-platform capability retention and new-platform adaptation. Project page: https://elispectre.github.io/UI-MOPD/.
Background
Background Analysis
1. Technical Context and Real-World Needs
Advances in multimodal foundation models (e.g., Qwen-VL, Claude) and agent systems have enabled GUI agents to automate tasks across digital environments (e.g., web navigation, mobile app interactions). However, real-world workflows often span heterogeneous platforms (computers, phones, web services), requiring agents to adapt to new environments while preserving platform-specific interaction conventions (e.g., closing a window on a computer vs. pressing "back" on a phone). The core challenge is: How can a single GUI agent continuously learn across platforms without losing its ability to interact effectively with each platform?
2. Limitations of Previous Approaches
Prior methods faced two critical bottlenecks:
- Data Scarcity: Existing datasets (e.g., OpenCUA) were often single-platform-focused, with noisy or misaligned actions, making cross-platform training unreliable.
- Behavioral Mixing: Simply combining data from multiple platforms led to "averaged" policies that confused interaction conventions (e.g., mixing computer scrolling with mobile swiping), causing catastrophic forgetting of platform-specific behaviors.
These issues prevented traditional approaches from balancing adaptation to new platforms and retention of existing capabilities.
3. Proposed Solution
The paper introduces UI-MOPD, a method that leverages multi-teacher on-policy distillation:
- Dynamic Teacher Selection: A "platform-specific teacher" is chosen based on the current environment (e.g., a computer teacher for desktop tasks, a mobile teacher for smartphone tasks), ensuring platform-appropriate behaviors are distilled into a shared policy.
- Behavioral Anchors: Each teacher acts as a stable reference, preventing the shared policy from mixing incompatible interaction patterns during learning.
Additionally, the team built Uni-GUI, a high-quality dataset with ~10K cross-platform trajectories, addressing data limitations.
4. Key Differences from Prior Work
Unlike traditional methods (e.g., mixed fine-tuning or model merging), UI-MOPD:
- Dynamically adapts to environments rather than statically aggregating data, avoiding behavioral confusion.
- Preserves platform-specific behaviors using teacher models as anchors, mitigating catastrophic forgetting.
Experiments show UI-MOPD achieves 38.2% and 12.0% success rates on OSWorld (computer) and MobileWorld (phone), outperforming baselines and demonstrating balanced cross-platform learning.
