Success is not stability. Two executions of the same task reach the goal: one stops after three actions; the other repeats a six-cycle loop. Follow the recorded evidence into a task-specific graph, step-level diagnosis, and full-trajectory metrics.
169 tasks ·10 applications ·2,909 GUI states ·3,718 evaluated trajectories
About the video, case validation, and source recordings
Coverage (0–8s) combines 12 recordings from ten apps. The case comparison (8–23s) shows all physical actions from two detailed-instruction Xiaohongshu-09 runs: Qwen3.8-27B (3 actions) and GPT-5.5 (15 actions). These are the same parameterized task with different live content, not a controlled head-to-head model ranking. Clips are action-centered and variably accelerated; the stable terminal frame is held. At 23s, one real loop is replayed at original speed, followed by graph alignment and calculations. The last 12 seconds animate all eleven models' detailed-instruction results from manuscript Table 2.
Every case action was visually checked against pre/post screenshots, temporal video evidence, and recorded coordinates. Both runs cover 3/3 required transitions. The unstable run has 6/15 deviations and 12/15 loop-covered actions. Its six one-step recoveries give RA=1; the stable run has no recovery opportunity, so RA is N/A. The original evaluator called the detour action “Back”; the actual click is the author entry. The film corrects that description without changing the validated state transitions or counts. The unstable agent reaches the goal at its action limit; it does not emit a clean completion signal.
Counts and metrics use the full trajectories, not edited excerpts. “ON / R” denotes an on-path action closing a recovery episode; loop membership is a separate sequence annotation. The full analysis contains 3,718 evaluated trajectories; 3,042 belong to the local video-validation population and 676 imported GUI-model records have no media. The opening montage does not imply simultaneous collection. Contact text is masked in the case footage.
In the unstable case, small masks cover author names and account identifiers while preserving the visible interactions.
Abstract
Why execution stability?
A clean execution and an unstable execution can satisfy the same semantic milestones. Final success does not tell us whether the agent took a detour, repeated actions, or recovered from a mistake. In GraceBench, 51.24% of successful trajectories still contain a deviation or loop.
Identical outcomes can hide different execution processes. GraceBench adds graph-aligned, step-level diagnosis.
The GraceBench framework
GraceBench constructs UI Transition Graphs (UTGs), curates task-specific subgraphs with alternative valid routes, and evaluates agents on physical Android devices. Synchronized action logs, screenshots, and video clips map each interaction back to the graph.
From application exploration to task curation and video-grounded trajectory diagnosis.
Alongside success rate (SR) and completion rate (CR), the benchmark measures deviation rate (DR), loop burden rate (LBR), and recovery through recovery ability (RA), recovery success rate (RSR), and average recovery latency (ARL).
Experimental results
Eleven models are evaluated on all 169 tasks under detailed and concise instructions, with a maximum of 15 actions per run. GPT-5.6 Sol achieves the highest success rate in both settings, but the lowest recovery ability on its observed recovery opportunities.
Detailed instructions · 169 tasks per model
Model
Best values are bold. Click a metric to sort. Rates are decimals; ARL is measured in steps. LBR is aggregated over trajectories with loops; recovery metrics are conditional on observed recovery opportunities.
Left: instability within successful trajectories. Right: stability differences between model pairs with matched task outcomes.
Benchmark coverage
The task suite spans Douyin, Dianping, Douyin Mall, Fanqie Novel, Amap, Hongguo, Qishui Music, TikTok, Toutiao, and Xiaohongshu. Ten reviewed application-level UTGs contain 2,909 canonical GUI states and 3,116 observed transitions.
Task domains and reference-path lengths. Each task has a concise goal and a detailed instruction variant.
We plan to release task definitions, instruction variants, UTG metadata, evaluation code, metric definitions, and trace schemas. See the arXiv preprint for the full protocol and human expert audit.
BibTeX
@misc{zhang2026gracebench,
title = {How GUI Agents Fail: GraceBench for Execution Stability Diagnosis},
author = {Ridong Zhang and Shuxian Liang and Chaoqun Wan and
Zhen Huang and Xian-Sheng Hua},
year = {2026},
eprint = {2610.19900},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2610.19900}
}
