WebFetch and WebSearch

What it is#

WebFetch and WebSearch are Claude Code’s two ways of reaching the live web from inside a session.

• WebFetch takes a URL, downloads it, extracts readable content (HTML to text/markdown, with the navigation chrome stripped), and returns it for the model to read. It is the right tool for “go look at this specific page” — a docs page the user just linked, a GitHub README, a changelog entry.
• WebSearch issues a search query and returns a list of result URLs with titles and snippets. It is the right tool for “find me a page about X” when you do not know the URL up front.

Both tools are read-only from the web’s perspective: WebFetch sends a GET, WebSearch sends a query to a search backend. Neither posts data anywhere. They are also content-aware: WebFetch does HTML cleanup so the model sees the readable body rather than the raw markup, which keeps token use sane.

When to use it#

• WebFetch — when you have a specific URL: documentation, a referenced blog post, a GitHub file’s raw URL, an RFC, a changelog. Also when the user pastes a link and asks “what does this say?”.
• WebSearch — when you need a page but do not have the URL. “Find the official docs for the X CLI”, “find a recent post about the Y bug”. Use the returned URL to feed WebFetch for the actual content.
• A pipeline of WebSearch → WebFetch — the common research move. Search to find the right page, fetch to read it.
• An Explore sub-agent — when the research is broader than two or three fetches. Delegating to a sub-agent keeps the parent’s context clean and lets the sub-agent run its own search-fetch loop without bloat.

Avoid both tools when:

• The information should be in CLAUDE.md or memory instead. Repeated fetches of the same docs URL is a sign the project’s brief is missing a key piece.
• The page is behind auth. Neither tool authenticates beyond what the host returns to an anonymous GET. Pages behind login flows return the login page, not the content.
• You need to interact with the page. Neither tool clicks, scrolls, or submits forms. For browser automation, a different tool (often outside Claude Code) is the right choice.

How it works#

WebFetch#

The model passes a URL and a prompt describing what it wants from the page. The tool:

1. Fetches the URL with a sane timeout and a sensible user-agent.
2. Parses the response. For HTML, it strips navigation, sidebars, and footers using readability-style extraction. For plain text, JSON, or markdown, it returns the body as-is.
3. Truncates to a per-call size budget (typically tens of thousands of characters) and returns the content along with metadata (final URL after redirects, content type, byte length).

The prompt argument lets the model narrow the request — “summarise the rate-limit section” — which can reduce how much of the page ends up in context. The fetched page itself is also cached for the session, so a second fetch of the same URL is cheap.

WebSearch#

The model passes a search query. The tool returns a list of results, each with a title, URL, and short snippet. The number of results is bounded (usually around ten); the search backend handles ranking. WebSearch does not fetch the underlying pages — for that, follow up with WebFetch.

Caching#

Both tools cache results in the session. The same URL fetched twice in one session returns from cache the second time, which means follow-up reads do not re-pay the network and parsing cost. Searches with identical queries similarly hit cache.

The cache is per-session. A new session does not inherit it. This is the right default — staleness matters, and pages change.

Limits#

• Per-call size. A single fetch is capped at a fixed character budget; the model gets the head and tail of the body if the page is larger. For very large pages, follow up with a more specific prompt or fetch a more specific URL.
• Rate. Both tools are throttled at the harness level. Bursting dozens of fetches in a few seconds can hit a backoff.
• Content type. Binary content (PDFs, images) is not extracted. For PDFs, use the Read tool with the file downloaded locally; for images, the dedicated image tools.

Configuration#

Allow/deny rules#

WebFetch and WebSearch can be allowlisted or denylisted via the permissions system:

{
  "permissions": {
    "allow": [
      "WebFetch(https://docs.example.com/**)",
      "WebSearch"
    ],
    "deny": [
      "WebFetch(https://internal.example.com/**)"
    ]
  }
}

The WebFetch rule pattern is a URL glob. A project that wants to scope research to specific docs hosts can do so explicitly; a project that wants to block the corporate intranet from leaving the air-gapped network can deny it.

Disabling entirely#

For sessions where outbound network is undesirable (offline work, regulated environments), both tools can be removed from the model’s tool list via the agent configuration. The model will route accordingly — it will not silently try to fetch and fail.

Examples#

Fetch a specific docs page#

{
  "tool": "WebFetch",
  "url": "https://docs.astro.build/en/guides/content-collections/",
  "prompt": "Summarise the schema validation section."
}

The tool returns the cleaned page contents along with the model’s summary intent. The model can then quote, paraphrase, or follow up.

Search then fetch#

// Turn 1
{
  "tool": "WebSearch",
  "query": "Astro 5 content collections breaking changes"
}

// Turn 2 — after seeing results
{
  "tool": "WebFetch",
  "url": "https://docs.astro.build/en/upgrade-astro/",
  "prompt": "Find anything about content collection breaking changes."
}

This is the everyday research pattern.

Pull a raw file from GitHub#

When you want the exact contents of a file in a public repo:

{
  "tool": "WebFetch",
  "url": "https://raw.githubusercontent.com/withastro/astro/main/packages/astro/src/content/loaders/glob.ts",
  "prompt": "Show me the glob loader implementation."
}

Raw URLs return source as plain text — no HTML extraction needed, and the content is more useful than the rendered page.

Delegate broader research to Explore#

{
  "tool": "Task",
  "subagent_type": "explore",
  "description": "Research a library",
  "prompt": "Find the recommended way to integrate library X with framework Y. Search docs, READMEs, and blog posts. Return a concise summary with citation URLs."
}

The sub-agent runs its own search-fetch loop. The parent gets the conclusion plus URLs.

Watch out for paywalled pages#

A blog post behind a paywall returns the teaser, not the article body. The tool does not error — it succeeds with reduced content. Sanity-check what the model says against the content actually fetched.

Gotchas#

• Auth-walled pages return the login page. Neither tool authenticates. If a page seems weirdly short or contains a “sign in to continue” banner, that is why.
• Truncation is silent. Large pages return a head and tail with a marker; the model may or may not notice. If a long page matters in full, fetch sub-sections separately.
• HTML extraction is best-effort. Pages with unusual structures (custom React/Astro single-page apps that render after JS) may return mostly empty. For SPAs, the fetched HTML often does not include the body content — the body is rendered client-side.
• Search results are snippets, not answers. The snippet may be misleading or stale; always WebFetch the top result before quoting.
• Cache is per-session. A long session that revisits the same docs page is cheap. A new session re-fetches.
• Rate limits are real. A burst of fifteen fetches in five seconds can hit backoff. Pace research, or delegate to a sub-agent that handles its own pacing.
• Do not fetch private URLs as a workaround. Pasting a URL with an embedded token (?token=abc) into a fetch is a credential leak — both into logs and into the model’s context. Use authenticated MCP integrations instead.
• Search backends drift. A query that returns the right page today may not next month. Hard-coding “the docs page is the third result” is fragile; the right move is to fetch and verify.
• No POST, no forms, no clicks. WebFetch is GET-only. For anything interactive, this is the wrong tool.

WebFetch vs WebSearch at a glance#

Related features#

• The Bash Tool
• Sub-Agents
• The Conversation Loop
• Context Window and Token Budget
• Permission Rules and Allowlists