In 83.73% of failed hidden-target runs, a GUI agent’s dominant failure was clicking the wrong thing that was already on screen. That number comes from a new arXiv preprint, “Chart the Invisible: Interactive Perception for Hidden-Target Exploration in GUI Agents” from Fudan University, which studied 7 GUI agents across 218 tasks and found the rate of correctly planned swipe actions 47.3 percentage points lower than click actions. One important scope note before anything else: this is a single, mobile-and-swipe-centric preprint from October 2026 (a date that derives from its arXiv identifier rather than a stated submission date), not yet independently replicated, and its headline system claims are self-reported. The share of real-world agent failures caused by hidden targets is unknown. What it does supply is a measured, specific failure mode that matches what many developers already see in browser and computer-use agent runs: the agent acts confidently on the visible state, and the visible state is not the state that matters.
If you build browser or computer-use agents with Playwright-style drivers, the practical consequence is this: a screenshot plus an accessibility-tree snapshot is not sufficient world state when the task target may be below the fold, inside a collapsed section, behind a hover menu, or simply not rendered yet. The fix is not a bigger model. It is an explicit perception policy: a bounded explore-then-recapture pass before acting on the best visible candidate.
The measured blind spot
The anchor study’s core finding is that exploration actions are planned far worse than direct actions. Quoting the paper:
“To quantify this difficulty, this paper conducts a study of 7 GUI agents across 218 tasks and finds that the rate of correctly planned swipe actions is 47.3 percentage points lower than that of click actions.”
The evaluated agents spanned general vision-language models (Qwen3-VL, GPT-5.4) and specialized GUI models (GUI-Owl, UI-TARS, GUI-G2, UI-Venus 1.5, MAI-UI), so the gap is not an artifact of one weak model family. And the failure analysis is the part that should change how you debug your own runs, the study finds:
“Further analysis reveals a dominant failure mode: in 83.73% of failed swipe tasks, agents decided to click on an incorrect visible target instead of exploring the correct but hidden region. Only fewer than 30% of failures involve selecting the wrong scrollable region or swipe direction.”
Read that carefully if you have been tuning scroll logic. The agents are mostly not bad at scrolling. They are bad at deciding to scroll at all. When the correct target is hidden, the best visible element becomes a trap, and the model takes it. As the paper puts it, “when the correct target is hidden, even the most relevant visible element may be an incorrect choice.”
Two secondary findings complicate any naive prescription. First, direction matters a lot: per the study, swipe planning averaged 57.4% for vertical swipes but only 25.3% for horizontal swipes, a pattern the paper describes as consistent with the vertical scrolling convention used for UI lists. Second, and more surprising, completely invisible targets were planned better than partially visible ones (47.2% vs 36.8%), and full-screen versus local swiping barely differed (44.6% vs 41.3%). The paper concludes region size is not the primary bottleneck. A partly glimpsed target, an ambiguous clipped row at the bottom of a container, is apparently worse than no hint at all, because it gives the model just enough to commit to the wrong thing.
This is a mobile study. Swipe conventions, small screens, carousel-heavy layouts. Transfer to desktop browser scrolling is unmeasured in these sources, so treat the exact numbers as directional, not portable.
A working taxonomy of hidden targets
“Off-screen” is not one problem. For a browser agent, the failure modes worth separating are:
| Hidden-target type | Why a single snapshot misses it | Pre-action check that catches it |
|---|---|---|
| Below-the-fold content | Outside viewport, may exist in DOM | Element-presence assertion against the tree, then scrollIntoView |
| Scrollable inner container (list, sidebar, table body) | Container scrolled, not the page; target in DOM but not visible | Scroll the specific container, not window.scrollTo |
| Carousel or horizontally paginated content | Only current slides rendered or visible | Horizontal scroll/swipe on the carousel, recapture |
| Collapsed sections, accordions, unopened tabs | Content not in the accessibility tree until expanded | Expand candidate containers, then re-query |
| Hover menus | Target only exists after hover/focus | Hover the parent, recapture, then click |
| Un-rendered state (lazy loading, virtualization) | Not in DOM at all yet | Scroll to trigger render, wait, re-assert presence |
That last row deserves emphasis. Virtualized lists render only the rows in view, so the accessibility tree is not merely incomplete about visibility; it is incomplete about existence. The agent’s “world” shrinks to whatever the front end happened to mount.
Hidden targets are also not an edge case in how these systems are trained. MagicGUI’s data pipeline explicitly includes instructions where, in the paper’s words, “a query such as ‘I want to check my rewards’ may not correspond to a directly visible ‘Rewards’ component on the current page but instead requires navigating through menus like ‘My’ or ‘Menu.’” Menu-mediated targets are a designed-for instruction category. And failure taxonomies keep naming the same problem independently: MobileUse’s AndroidWorld analysis defines six failure types and includes “Navigation failures, where the agent struggles to find a certain element or function, suggesting deficiencies in layout understanding and navigation.”
Why screenshot plus accessibility tree is not sufficient state
The standard browser-agent substrate, exemplified by Microsoft’s Playwright MCP, “exposes a headless browser controlled via accessibility tree snapshots instead of raw screenshots,” with vision as fallback “for visually rendered content that lacks semantic structure.” That is a sound default. It is fast, token-cheap, and precise for semantic elements. But both channels share a blind spot: they capture what is rendered now.
The tree omits un-rendered and collapsed content. The screenshot omits everything outside the viewport. Neither tells you whether the current view is the complete searchable space for the task. The agent then faces what the anchor paper frames as a forced decision under uncertainty: “At every step, GUI agents are thus forced to decide under uncertainty whether to pick a visible target or explore hidden areas.” Give the model only a snapshot, and you have silently answered that question for it: pick visible.
Worse, the snapshot decays. EvoSkill-GUI, relaying findings from deployment studies, reports that “a skill valid at one interface state may fail when a pop-up appears, a widget shifts, or an accessibility tree becomes stale, and idempotent failure loops account for a substantial portion of timeouts in real GUI deployments.” A cached or stale tree does not just miss hidden targets; it confidently describes a UI that no longer exists, and the agent loops.
One more caution against the “throw a better model at it” instinct: UI-Evol’s OSWorld experiments found that even 90%-correct retrieved knowledge yielded only a 41% task success rate. Better information about what to do does not fix a perception pipeline that cannot see what is there. The browser-agents paper similarly argues that architecture, not model capability, limits agent performance, though that is one author team’s claim and should be read as such.
Exploration policy: scroll, expand, hover, recapture, and when to stop
The counterintuitive implication of the anchor study is that agents need a trigger, not a technique. They can scroll; they fail to decide to scroll. So the policy question is: when should the agent distrust the visible state enough to spend a step exploring?
A reasonable set of triggers, bounded to what the evidence supports:
- Presence assertion fails. The expected target (or any plausible match) is absent from the tree and screenshot. Explore before declaring failure.
- Best visible match is weak or ambiguous. This is the 83.73% trap: a plausible-but-wrong candidate is available. The paper’s own data shows even the most relevant visible element may be incorrect when the true target is hidden, so a low-confidence visible match should buy at least one exploration step, not a click.
- The container structure implies more content. Scrollable regions, pagination, carousels, collapsed sections: their existence is visible evidence of hidden state.
For the exploration itself, the most concrete open-source guidance comes from GUI-explorer’s error-handling protocol, quoted here verbatim:
“2. Error Handling Protocol:
- Switch approach after ≥2 failed attempts
- Prioritize scrolling (‘scroll’ action) over force-acting on invisible elements
- If an element is not visible, use ‘scroll’ in the likely direction (e.g., ‘down’ to find elements below the current view).
- Try opposite scroll direction if initial fails (up/down, left/right)”
That protocol encodes exactly the right shape: scroll before force-acting, retry the other direction, and cap the whole thing. Directional scroll is a first-class action primitive in standard mobile action spaces: scroll(direction='down or up or right or left') is UI-TARS’s action signature, mapped in the AgentCPM-GUI appendix onto that system’s own normalized primitive, to, which takes a direction (reversed, because the two systems define scroll directions oppositely). In Playwright-style browser drivers the equivalent is scrollIntoView or container-scoped wheel events, so none of this requires exotic tooling.
The cap is not optional. Exploration without a budget converts wrong-click failures into timeout loops, the failure mode that deployment studies cited by EvoSkill-GUI tie to a substantial portion of timeouts in real GUI deployments. The budget framing generalizes beyond scrolling: EcoAgent-Bench describes the need for “two signals that current agents often conflate: whether the evidence is sufficient and whether the next action is affordable.” Every scroll, expand, and recapture costs steps and latency. And step budgets are real in evaluation too: a human audit of all 165 WebArena-Lite tasks evaluated agent trajectories under explicit 15- and 25-step action budgets and built a first-consequential-error taxonomy that separates navigation failures from other error types. Spend your steps where they buy information.
What the new preprint proposes, and what remains unproven
The paper’s proposed system, Hagasu, provides scrollable-region cues derived directly from the current screenshot, essentially telling the model there is more content here and it lives in this region, so the explore decision is made on evidence rather than guesswork. The authors report it is effective with modest overhead.
Hold that claim at arm’s length. The gains are self-reported within the paper’s own evaluation, on a mobile-centric task set, in an October 2026 preprint that has not been independently replicated. The idea, surface the existence and location of hidden regions in the perception input itself, is well-motivated by the failure analysis. Whether Hagasu’s specific implementation delivers its reported gains, and whether the approach transfers to desktop browser scrolling conventions, is unmeasured.
There is also a competing bottleneck you should not conflate with this one. MEGA-GUI documents that “all Vision-Language Models (VLMs)—regardless of whether they are fine-tuned for grounding tasks—struggle to accurately localize target GUI elements in a single pass,” with targets on dense screens sometimes occupying under 0.1% of screen area. Its training-free multi-stage zoom reaches 73.18% on ScreenSpot-Pro and 68.63% on OSWorld-G by improving grounding on targets that are already fully visible. That is a different problem with a different fix. An agent can have excellent hidden-target exploration and still miss the right pixel, or nail grounding and still click the wrong visible decoy because the real target was never revealed. If your runs fail, diagnose which one you have before choosing a remedy; the symptoms look identical in a pass/fail log.
Also worth knowing: Browser Use reports 89.1% success on WebVoyager across 586 tasks (WebVoyager itself reports roughly 59%), per the browser-agents paper. That benchmark’s hidden-target coverage is unstated in these sources, so this neither confirms nor refutes the blind spot. It mostly shows the gap can hide inside aggregate scores.
Practical verdict
Before clicking the best visible candidate, treat one screenshot or accessibility snapshot as insufficient state. The recipe I would implement, as an inference from the mobile measurements plus the prompt practice of open-source GUI agents rather than a directly evaluated result (none of these sources reviewed here measure this pipeline end-to-end on web agents):
- Assert presence of the target in the accessibility tree and/or screenshot.
- If absent or ambiguous, explore the likely container: scroll it vertically first (the 57.4% vs 25.3% direction gap argues for vertical priority), then the opposite direction, expanding collapsible sections and hovering menus along the way.
- Recapture the tree and screenshot after each exploration action. Never act on a pre-exploration snapshot.
- Cap the pass: switch approach after two failed attempts, per GUI-explorer’s protocol, to avoid the idempotent timeout loops that deployment studies link to real GUI timeouts.
The cost is real: extra steps, extra latency, extra tokens per task, and none of these sources quantify that overhead on Playwright-style browser stacks. I would accept that cost on flows where a wrong click is expensive, like mutating actions, checkouts, form submissions, and trial the recipe on tasks that can tolerate retries first. If a misclick means charging a customer or submitting the wrong record, the measured alternative is an agent that picks a visible-but-wrong target in the large majority of failed hidden-target runs.
The honest limits: the quantitative core is one unreplicated preprint, 218 tasks, 7 agents, mobile-first. The share of real-world web-agent failures caused by hidden targets is unknown, and grounding precision on dense visible screens is a documented parallel bottleneck that exploration does nothing for. What is well-supported across independent sources is narrower and more durable: snapshots go stale, visible best-matches are traps when targets are hidden, at least one open-source GUI agent already encodes a scroll-first protocol, and the decision that fails is usually whether to look, not how. That decision is the one worth engineering.
Frequently Asked Questions
What is the dominant failure mode for GUI agents when the target is hidden?
Further analysis reveals a dominant failure mode: in 83.73% of failed swipe tasks, agents decided to click on an incorrect visible target instead of exploring the correct but hidden region. Only fewer than 30% of failures involve selecting the wrong scrollable region or swipe direction.
Join the discussion
Share a useful perspective or ask a question about this article.