WebVoyager — Multimodal Web Agent

Context#

For a decade, “web agent” research was a story of brittle DOM parsing. You’d point a system at a website, extract the accessibility tree or a flattened HTML, hand the agent a list of clickable elements identified by some kind of opaque ID, and hope the page didn’t change overnight. The agents worked on training-set sites and shattered on novel ones. Real websites are visual artefacts — they communicate through layout, hierarchy, colour, and proximity — and a text-only agent was reading them with one eye closed.

WebVoyager, published in 2024, took the obvious-in-hindsight step: give the agent a vision-language model that can see the rendered page directly. Instead of “click element 47,” the agent sees a screenshot, decides “click the blue Sign In button in the top right,” and the system finds and clicks it. The result is an agent that can drive real, unmodified websites — shopping, booking, search, account management — end to end.

The interesting bit isn’t that screenshots work better than DOMs (in retrospect, of course they do). The interesting bit is the grounding mechanism — how the agent’s natural-language pointer (“the blue button”) becomes a coordinate the browser can click, and how the system survives the messy reality of pages that scroll, lazy-load, popup, and re-flow on every visit.

Problem#

The concrete problem WebVoyager attempts to solve:

• Input. A user task in natural language (“book me a flight from London to Tokyo, leaving next Friday, returning in two weeks, economy, under £900”).
• Output. A completed task on a real website — the action taken, screenshots of the trajectory, and either the final result (booking confirmation) or a structured failure report.
• Constraint. No site-specific scaffolding. The agent has to drive any website given a URL.
• Hard parts.
  • Visual grounding. Translating “the blue Sign In button” into a (x, y) coordinate on a page that may be 4000 pixels tall and 1920 wide.
  • Dynamic content. Modals, cookie banners, auto-refreshing layouts, lazy-loaded panels. The agent’s view of the page is stale the moment it acts.
  • Multi-step task completion. Booking a flight is 8–15 actions; agents drift over that horizon.
  • Recovery from wrong actions. Pages move on; “go back” doesn’t always restore state.

Architecture#

WebVoyager is a multimodal ReAct loop with three components: a vision-language model (the agent’s brain), a browser controller (Playwright or Selenium with custom instrumentation), and a screenshot-and-annotation pipeline that bridges them.

   ┌─────────────────────────────────────────────────────────────┐
   │                       User Task                              │
   │           "Book me a flight from London to Tokyo..."         │
   └──────────────────────────────┬──────────────────────────────┘
                                  │
                                  ▼
   ┌─────────────────────────────────────────────────────────────┐
   │                    Multimodal Agent (VLM)                   │
   │   Inputs each turn:                                          │
   │     - task                                                   │
   │     - screenshot with overlaid interactive-element marks     │
   │     - text list of marks (id, role, name)                    │
   │     - action history                                         │
   │   Output each turn:                                          │
   │     - Thought                                                │
   │     - Action: click/type/scroll/wait/answer                  │
   │     - Target: mark-id (resolved to coordinates)              │
   └──────────────────────────────┬──────────────────────────────┘
                                  │
                                  ▼
   ┌─────────────────────────────────────────────────────────────┐
   │              Action Resolver and Executor                   │
   │   - Map mark-id back to (x, y) coordinates                  │
   │   - Execute via Playwright (click/type/scroll)              │
   │   - Wait for DOMContentLoaded + idle                        │
   └──────────────────────────────┬──────────────────────────────┘
                                  │
                                  ▼
   ┌─────────────────────────────────────────────────────────────┐
   │            Observation: Screenshot + Annotation             │
   │   - Take screenshot of viewport                              │
   │   - Run mark-extractor (interactive elements)                │
   │   - Overlay numbered boxes on the screenshot                 │
   │   - Build text list mirroring the boxes                      │
   └──────────────────────────────┬──────────────────────────────┘
                                  │
                                  └─── loop until: done | budget hit

Grounding by Set-of-Mark#

The crucial mechanic is Set-of-Mark prompting, popularised by Microsoft Research and adapted by WebVoyager. After each action:

1. The browser takes a screenshot.
2. An extractor identifies every interactive element on the page (links, buttons, inputs, selects, role=“button” elements) and gets each one’s bounding box.
3. The system draws a numbered box on the screenshot — 1, 2, 3, … over each element.
4. A parallel text list is built: mark 1: button "Sign In", mark 2: input "Email", ....

The agent sees the annotated screenshot and the text list together. When it decides to act, it produces “Click mark 7” — a discrete, well-typed action. The system then maps mark 7 back to its bounding box and clicks the centre.

This is the difference between asking the model to produce coordinates (unreliable) and asking it to pick an integer from a small set (reliable). The visual annotation lets the model use vision; the parallel text list gives it a structured handle.

Action space#

Five actions cover the vast majority of web tasks:

• click(mark-id) — click an element.
• type(mark-id, text) — focus an input and type.
• scroll(direction) — scroll the viewport up/down, also page-down/up.
• wait — wait for the page to settle (used after slow loads).
• answer(text) — terminate the task with a final answer (used for question-answering tasks where the goal is information retrieval, not state change).

A sixth action — go_back — handles wrong-page recovery. Notably absent: typing free-form coordinates, drag-and-drop, hover-only menus. WebVoyager’s coverage stops where the action space stops.

Key innovations#

What makes WebVoyager work beyond “a multimodal model in a browser loop”:

1. Set-of-Mark as the grounding interface. The combination of visual numbered boxes plus a parallel text list is the unlock. The agent can use vision (to see what’s where) and language (to read element names) without having to reason in pixel coordinates. The mark IDs are the abstraction barrier.

2. A bounded discrete action space. Five primary actions, each with a small parameter shape. The agent never produces malformed actions because the prompt and the schema constrain it to the set. Recovery from bad pages comes from picking different marks, not from producing different action shapes.

3. The annotated screenshot is the whole observation. No DOM tree in the prompt, no accessibility tree dump. The annotation pipeline does the work of extracting “what’s interactable” so the model only sees what it can interact with. This is a much cleaner separation of concerns than the prior generation of agents that mixed DOM dumps with screenshots.

4. Robust observation pipeline. Pages take time to load; the system waits for idle, retakes the screenshot, and re-extracts marks before handing back to the agent. This is unglamorous infrastructure — and it’s what separates a demo from a system that works on real sites.

5. Trajectory logging for offline analysis. Every turn — screenshot, mark list, agent reasoning, chosen action, resulting page — is logged. Failures can be replayed and inspected. This is the part most homegrown web agents skip and regret.

Evaluation#

WebVoyager was evaluated on a curated benchmark of 15 real websites (popular consumer sites like Amazon, Coursera, BBC News, ArXiv, ESPN, Google Search) with around 40–50 tasks per site. Tasks were a mix of information-retrieval (“find the most-cited paper by author X”) and state-change (“add product Y to cart”).

Headline results from the paper:

• End-to-end task success. WebVoyager reportedly hit 50–60% success rate across the benchmark, with site-by-site variance — some sites (search-heavy) much higher, others (booking flows) lower.
• Substantial gains over text-only baselines. Earlier text-only agents on similar benchmarks were at 20–30%. The vision-language model is the headline contributor.
• Open vs closed models. The paper compared a closed-source vision-language model (top-tier API) with open-weights vision models; the closed-source model was meaningfully ahead. This gap has been closing but was real at publication time.

The evaluation uses human raters to grade each trajectory against a rubric — did the agent reach the goal? did it take only correct actions? This is expensive but unavoidable for end-to-end web tasks; there is no programmatic ground truth.

Trade-offs and limitations#

Other limitations:

• Cost stacks fast. A 10-action trajectory on a vision-language model costs an order of magnitude more than a text-only agent. Production-shaped use needs aggressive screenshot compression and selective re-observation.
• Mark extraction is the failure point you didn’t anticipate. If the extractor misses an interactive element, the agent can’t use it. Sites that hide interactivity behind unusual roles or off-screen panels cause silent failures.
• No within-page state. The agent sees only the current screenshot. If a key piece of information was on a previous screen and is no longer visible, the agent loses it. Some implementations add a short-term memory of “facts I extracted from past screenshots”; the original paper had limited memory of this kind.
• Anti-bot detection. Real websites increasingly fingerprint automated browsers. WebVoyager’s research setup doesn’t deal with this; production deployments have to.
• Reproducibility. Real websites change. A benchmark trajectory from last year might be impossible to replicate today because the site re-flowed. The community handles this with snapshot-based benchmarks (Mind2Web, VisualWebArena) that freeze the page state.

Lessons#

Generalisable takeaways from WebVoyager:

• Match the input modality to the environment’s communication channel. Websites communicate visually. A vision-language agent reads them more like a user does. Forcing a vision-rich environment through a text-only pipeline throws away information.
• Discretise the action space aggressively. Letting the model produce free-form coordinates or free-form CSS selectors is the road to debugging hell. A small discrete action vocabulary with typed parameters is easier to debug, easier to constrain, and easier to fail safely.
• The observation pipeline is half the system. Most web-agent failures aren’t “the model picked wrong”; they’re “the model picked from a stale or incomplete observation.” Get the screenshot-and-annotate loop right before tuning prompts.
• Log every trajectory. You can’t improve a web agent you can’t replay. The trajectory log is the dataset for prompt tuning, the dataset for fine-tuning if you go there, and the debug surface for individual failure investigations.
• Don’t fight the page. Sites have idle states and animation cycles. Waiting for those is faster than trying to act through them. Build the wait-and-confirm loop into the executor.

Related systems#

• Building a Multimodal Web Agent with ADK
• The ReAct Loop
• Perception and Grounding
• MuLan — Multimodal Diffusion Agent