Video walkthrough

WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites

Overview 3:40 CC AI

paperi.ai
0:00 / 0:00

Jiangang Han

An AI can understand exactly what button to press and still fail, because the click lands somewhere else. This paper finds that the invisible machinery between the AI and the website can be as important as the AI’s reasoning.

Abstract

We present WebFovea, a vision-based web agent that placed 2nd in the WebRetriever Challenge 2026 [1, 2] with a final score of 57.0 out of 100. The challenge evaluates agents end to end on Protocol III of the WebRetriever benchmark [1]: starting from an entry URL on a live website, the agent must operate the site’s own interface and return a verifiable answer. A capable multimodal large language model (LLM) is necessary for this, but not sufficient. The model’s decisions reach the browser through the harness, the code between the model and the page. At every step, four things must go right: the model’s reply must be parsed into the intended action, the action must take effect on the page, the result must be reported back accurately, and the model must be shown the information it needs. On real websites, many of the failures we observed occurred at one of these four stages rather than in the model’s reasoning. A coordinate-space mismatch placed every click at 3/4 of its intended coordinates; actions on native dropdowns, inside iframes, and in text boxes failed silently; and self-generated chat-template tokens contaminated 4.9% of task episodes. WebFovea hardens each stage and surrounds the loop with guardrails that keep the agent within the rules and its budget. The four-stage view does not depend on the model, although some individual fixes do. Because we used the same model in all four submissions, the rise of our official hidden-set score from 31.0 to 57.0 reflects changes to the harness, up to run-to-run variance on live sites. We describe the design, the evidence for each component (including negative results), a failure analysis, the limitations, and a roadmap that includes routing different steps to different models. Code: https://github.com/jianganghan/WebFovea Much of the information people and organizations need lives on public websites: statistical portals, regulatory filings, rankings, and dashboards. Retrieving it usually means navigating menus, setting filters, reading tables and charts, and only then reading off a value. Web agents that automate this are useful only if their answers can be trusted. The WebRetriever benchmark [1] evaluates such agents on 1,550 tasks on 800 real, live websites. Its Protocol III, the challenge track, asks for end-to-end retrieval: navigate from an entry URL, operate the page, and return a single answer that can be checked exactly. Success requires both a correct answer and a trajectory that actually reaches the evidence. The benchmark’s own results show why both are needed: navigation success alone is an insufficient predictor of task completion [1]. • Answer-level grounding. The answer must come from the authoritative page, reached through its interface during this session, not from parametric memory, a search engine, or a hand-crafted API call. Our logs show why this matters. After hitting a CAPTCHA, an early version of our agent answered from memory; in another task, the model wrote an entire fictitious multi-step trajectory in a single reply and submitted an answer at step 1 (§4.2). Web data also drifts: by the time we evaluated, some of the 100 public reference answers no longer matched the live sites (§5.5). In such cases, only the trajectory and the final on-screen evidence can show whether the agent’s answer was right. • Action-level grounding. Every action must land on the intended pixel or element, and its effect must be perceived correctly. For a vision-based agent, this is where many of our failures turned out to lie. Both levels depend on the harness: everything between the model and the browser. We find it useful to view each step as a round trip that passes through four stages (Figure 1, §4.1). The model’s reply is turned into an action (parsing), the action is carried out on the page (execution), the page’s reaction is reported back (feedback), and the next view shows the model what it needs (observ

Transcript

An AI can understand exactly what button to press and still fail—because the click lands somewhere else. This paper finds that the invisible machinery between the AI and the website can be as important as the AI’s reasoning. Much of the information people and organizations need lives on public websites, from statistical portals and regulatory filings to rankings and dashboards.

Retrieving it usually means navigating menus, setting filters, reading tables and charts, and only then reading off a value. Web agents that automate this are useful only if their answers can be trusted. The challenge asks an agent to start from an entry page on a live website, operate the site’s own interface, and return an answer that can be checked.

A capable language model is necessary, but not sufficient, because its decisions reach the browser through the code between the model and the page. At every step, four things must go right: the reply must become the intended action, the action must take effect, the result must be reported accurately, and the model must see the information it needs.

A useful way to picture each step is as a round trip, like sending a friend directions and waiting for an accurate report of what happened. The model’s reply is turned into an action, the action is carried out on the page, the page’s reaction is reported back, and the next view shows the model what it needs.

When one stage breaks, even a strong model can click empty space, retype text that never arrived, or abandon an action that actually worked. The model is right, but the click is wrong. The fix maps coordinates by the ratio between the image sent to the model and the actual page, measured while the agent is running.

The model had been accurate to within a few pixels all along; the error was entirely in the conversion. The two views make the mismatch visible: the original click lands beside the intended button, while the corrected click sits directly on it.

The later evaluation improved mainly because the agent finished more tasks, rather than because it answered more accurately on each finished task. Roughly seven of the ten points came from completing more tasks, and about three came from answering more accurately.

The strongest component evidence showed that the added hover behavior solved four of five tasks needing tooltip values. Raising the step limit increased the number solved from zero or one to four, and two of those four needed more than the earlier limit. The reload budget recovered tasks stuck on loading pages, saved budget in others, and hurt none.

The component evidence came from small targeted sets and shows direction, not statistical significance. Because live sites change and some were unreachable, repeated runs of the same version vary, and each official score came from a single run. When a vision-based web agent fails on a live website, the cause often lies not in its reasoning but in how its decisions are carried out and reported back.

Making those stages reliable raised the challenge score from 31.0 to 57.0 across four submissions and placed the agent second. For anyone building or relying on such tools, the practical lesson is simple: check that decisions reach the page, and that the page’s response reaches the model.

Reliable web agents need a dependable round trip: the decision must become the intended action, the page must respond, and that response must reach the model accurately. Fixing that path raised performance even without changing the model.

A derivative work by Paperi · AI-generated script, voice and captions · pages and figures unaltered

Made with Paperi.

Drop in a research PDF — get a narrated video walkthrough like this one, with highlights that follow the narration. Free to start.

Try it with your paper →

More in Computer Science

Grounding with Confidence: Controllable Generative Video Temporal Grounding 4:37

Grounding with Confidence: Controllable Generative Video Temporal Grounding

A video search system can find a clip that sounds right, but still has no idea whether it should trust that answer. This paper gives each proposed clip a separate confidence score, including permission to return nothing.

The Score Is Not the Structure: Brain Alignment and Cross-Lingual Transfer 4:48

The Score Is Not the Structure: Brain Alignment and Cross-Lingual Transfer

A model can score as if it matches the brain even after the brain information has been destroyed. And a language pattern can look real when the test measuring it is failing.

FreeLoc: Online Floorplan Localization via Diffusion-Aided Pose Refinement 4:04

FreeLoc: Online Floorplan Localization via Diffusion-Aided Pose Refinement

A robot may have a floorplan, yet still need a huge prepared database before it can tell where it is. FreeLoc asks a simpler question: can the robot query the map directly, as it moves?

All 21 papers in Computer Science →