A web agent can reason correctly and still fail because its click lands in the wrong place. WebFovea argues that reliable browsing is less about smarter thoughts alone and more about making the model-to-browser round trip dependable.

Much of the information people and organizations need lives on public websites, including statistical portals, regulatory filings, rankings, and dashboards. Retrieving that information usually means navigating menus, setting filters, reading tables and charts, and only then reading off a value.

Web agents that automate this are useful only if their answers can be trusted. The WebRetriever benchmark evaluates agents on 1,550 tasks across 800 real, live websites, and its Protocol III challenge asks for end-to-end retrieval from an entry URL to a checkable answer.

Success requires both a correct answer and a trajectory that actually reaches the evidence, because navigation success alone is an insufficient predictor of task completion. WebFovea is a vision-based web agent that placed second in the WebRetriever Challenge 2026 with a final score of 57.0 out of 100.

In Protocol III, the agent starts from an entry URL on a live website, operates the site’s own interface, and returns a verifiable answer. A capable multimodal large language model is necessary, but not sufficient. The model’s decisions reach the browser through the harness, the code between the model and the page.

At every step, the reply must be parsed, the action must take effect, the result must be reported accurately, and the model must see the needed information. Many observed failures occurred at those four stages rather than in the model’s reasoning: a coordinate mismatch placed every click at three quarters of its intended coordinates, while several actions failed silently.

WebFovea hardens each stage and surrounds the loop with guardrails, and the same model’s official hidden-set score rose from 31.0 to 57.0 as the harness changed. Answer-level grounding means the answer must come from the authoritative page reached through its interface during the session, not from parametric memory, a search engine, or a hand-crafted API call.

The logs include an early version answering from memory after a CAPTCHA, and another task where the model wrote a fictitious multi-step trajectory in one reply and submitted an answer at step 1. Web data also drifts: by evaluation time, some of the 100 public reference answers no longer matched the live sites.

In those cases, only the trajectory and the final on-screen evidence can show whether the answer was right. Each Protocol III task gives a natural-language instruction and an entry URL. The agent must operate the live site and call finished, with its answer as the content.

The tasks span document extraction, form interaction, multi-source comparison, complete data retrieval, and chart analysis. There is no partial credit, and final scores combine answer correctness with manual review of trajectories and code. At every step, the model looks at what it is shown and decides on one action.

The model never touches the browser directly: the harness carries its decision to the page and carries the page’s reaction back. WebFovea views this as a round trip through four stages, and a step succeeds only if all four work. The figure’s center shows the four-stage loop as two linked paths: parsing leads into execution, while the page’s change returns through feedback into observation.

The labeled details make the harness concrete: it strips self-generated tokens during parsing, uses coordinate alignment and DOM fallback during execution, and combines fingerprints, read-back, and tooltip hashes for feedback. Look across the middle column to see that the failures span the entire loop, from malformed model output during parsing to missed hover-only values during observation.

The right column pairs each failure with a targeted safeguard, including coordinate alignment, frame fingerprints, text-finding tools, and executor budgets. WebFovea keeps the observe–think–act loop of the reference implementation.

At each step, the model receives the task, the action space and hard constraints, the full text history, the five most recent screenshots, and, when relevant, a one-sentence feedback note. The model replies with a Thought and one Action. It receives no DOM or accessibility tree and acts through pixel coordinates.

DOM access is confined to the executor and two narrow text channels, feedback notes and read text. A single executor function is the only place where the agent can affect the browser. The coordinate fix downsizes every screenshot to at most 1440 by 810 and maps coordinates by the ratio of the sent image to the actual viewport, measured at runtime.

On a real page, three targets previously missed by 122 to 189 pixels were hit within 1 to 5 pixels. The model had been accurate to within a few pixels; the error was entirely in the conversion. On a paired set of 24 development tasks, success rose from 8 out of 24 to 20 out of 24, with 12 tasks fixed and none broken.

The share of tasks that hit the step limit fell from 67 percent to 4 percent, and the version containing the fix also included other changes. The figure gives a direct visual comparison: in panel a, the red click lands above and left of the “Movie” option, while in panel b, the red click is centered on that option.

The ring makes the contrast especially clear—the same intended target is missed before alignment and reached after the coordinate scaling. Before and after every action, the executor records a DOM fingerprint containing the URL, title, node count, text length, scroll offsets, and focus, then compares the two.

Unlike a pixel difference, the fingerprint is not fooled by ticking clocks or rotating carousels. If nothing changed, the model receives a note naming the element under the click point and whether it is clickable. The fingerprint originally covered only the main frame, so a successful click inside an iframe was reported as no detectable change and the model abandoned an action that had worked.

WebFovea also fingerprints the child frame, ignores focus moving into an iframe as a change, and compares a hash of page text for hovers and SVG or canvas targets. Actions report their own results too: fill reads back the box’s actual value, while a failed select returns the available options.

Development used the 100 publicly released Protocol III tasks with the organizers’ explicit permission, split deterministically into dev70 and holdout30. Answers were scored locally with exact, numeric matching using a 0.5 percent relative tolerance, and date matching, plus an optional language-model judge for semantic equivalence.

Sixteen of the 100 sites were unreachable from the network, leaving 25 scorable tasks in holdout30. On the same code version, those 25 tasks scored 52.0 percent locally, while the hidden set scored 46.0 percent officially. The 6-point gap was within one binomial standard deviation, about 10 points for a sample size of 25, so local numbers were treated as a reasonable guide for relative comparisons.

Table 4 lists four official submissions on the hidden set, but each submission bundled several changes, so gains cannot be attributed to single components. The plus 10 at the second submission came mostly from finishing more tasks.

The first run finished only 79 of 100 tasks before its time limit. At its own accuracy of 31 out of 79, or 39.2 percent, finishing 97 tasks as the second run did would have scored about 38. Roughly 7 of the 10 points came from completing more tasks, and about 3 came from answering more accurately.

The score column shows a steady progression across the four dated submissions, reaching fifty-seven point zero on September second. The additions column associates that final row with iframe fallback, searching child frames, an eighty-step limit, and model-call retries; the table also notes that manual review deducted no points.

Table 6 groups all 100 local tasks by their best outcome across all runs, so the count of correct answers is an upper bound for any single version. Many remaining errors are not the agent’s: in 13 public tasks, the reference answer no longer matched the live site because data had changed or the metric definition differed.

Sixteen sites were unreachable from the network. The agent’s own errors were either capability gaps, such as canvas-only widgets and dense charts, or decision errors that a verifier and task-type knowledge might address. For maintaining WebRetriever and similar live-site benchmarks, the suggested safeguard is to record each reference answer with a timestamped evidence screenshot.

The table’s right-hand Tasks column shows the distribution across six outcome buckets. After the approximately fifty-five correctly answered tasks, the largest explicitly counted categories are unreachable sites at sixteen and outdated or differently defined references at thirteen; structural obstacles appear in six tasks, while capability gaps and wrong-path decisions are listed at approximately four and three.

WebFovea’s central lesson is that a vision-based web agent succeeds only when parsing, execution, feedback, and observation all work together. Fixing that harness helped move the official hidden-set score from 31.0 to 57.0, while the remaining errors expose real limits of live-web evaluation.