[N13a] Add a web_fetch built-in with pinned-DNS SSRF checks and a redirect cap

[LinuxDevil]

Goal

Agents can read a web page with one built-in tool, web_fetch, that is safe to hand to a model: it only does GET, refuses private and loopback destinations on every hop (with the connection pinned to the checked address, so DNS rebinding cannot bypass the check), caps redirects, response size and time, and returns readable text instead of raw HTML. AUDIT-2 row "Web tools" (eve: built-in web_fetch "with SSRF checks and 10-redirect cap"; us: the http tool only). Security-sensitive, hence model:opus.

Current state

• The http tool: src/tools/built-in/http.ts. createHttpTool(options) (line 406) builds http_request ({ url, method, headers?, body? }); HttpToolOptions (lines 14-32): timeout (30000), maxRedirects (5), validateSSL (true). No host allow list, no response size limit (readResponseBody reads the whole json() or text(), lines 266-277).
• Its SSRF check: BLOCKED_RANGES (lines 44-53: 127/8, 10/8, 192.168/16, 172.16/12, 169.254/16, ::1, fc00::/7, fe80::); isBlockedHost() (lines 127-143) runs dnsPromises.lookup(bare, { all: true }) and then the request is made by a separate fetch (createDirectTransport, line 173: fetch(url, init) when TLS validation is on), which resolves the name again. The check is therefore not pinned to the connection: a name that resolves to a public address for the check and to 127.0.0.1 for the connection passes. src/deploy/runtime.worker.ts:30-31 describes this tool as "DNS-rebinding-safe resolve-then-verify" with "a pinned undici Agent/dispatcher", and docs/tools.md:123 says "every resolved address is checked before connecting"; neither matches the code. Redirects are followed by hand with a re-check per hop (fetchFollowingRedirects, lines 241-263).
• A pinned, more complete check already exists for the credential broker: src/security/credentialBroker.ts PRIVATE net.BlockList (lines 104-108: 0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.168/16, 224/3; ::/127, fc00::/7, fe80::/10, ff00::/8), inList() unwrapping IPv4-mapped IPv6 (lines 111-115), and admit() (lines 142-155), which returns the checked address to dial.
• undici is a regular dependency (^8.11.2), so an Agent with a custom connect.lookup is available without a new dependency.
• Built-in tools are exported via src/tools/built-in/index.ts -> src/tools/index.ts -> root. Spec files can name built-ins through RESOLVABLE_BUILT_IN_TOOLS (src/spec/specToAgent.ts:59-63: http, current-date, day-name).
• defineTool options include annotations? (src/tools/defineTool.ts:11-47). Grep web_fetch|webFetch in src/: no hits.

Scope

In:

• New shared module src/security/privateAddress.ts (Node only): isPrivateAddress(address: string): boolean built from the broker's list plus 192.0.0.0/24, 198.18.0.0/15, 64:ff9b::/96 and 2002::/16 (NAT64 and 6to4 can reach IPv4 private space), with IPv4-mapped unwrapping; and pinnedLookup(options: { allowPrivate?: readonly string[] }), a dns.lookup-compatible function for undici's connect.lookup that resolves all addresses, throws SsrfBlockedError when any is private (unless the host matches allowPrivate via matchesHost() from src/security/hostPattern.ts), and returns the checked address so the socket connects to exactly what was checked. Make credentialBroker.ts import isPrivateAddress instead of its local PRIVATE list (behavior unchanged for the broker except the four added ranges; its tests must stay green).
• New tool src/tools/built-in/webFetch.ts, exported as webFetchTool and createWebFetchTool(options):

  export interface WebFetchToolOptions {
    timeoutMs?: number;            // default 30_000, whole request including redirects and body
    maxRedirects?: number;         // default 10 (as eve); each hop re-resolved and re-checked by pinnedLookup
    maxBytes?: number;             // default 2 MiB read from the network, then stop reading
    maxChars?: number;             // default 50_000 characters returned to the model
    allowedHosts?: readonly string[];   // host patterns; when set, anything else is refused before DNS
    blockedHosts?: readonly string[];
    allowPrivate?: readonly string[];   // host patterns allowed to resolve to private addresses (e.g. an intranet docs host)
    userAgent?: string;            // default 'lousho-web-fetch'
  }
  // input:  { url: string }  (http: or https: only, no credentials in the URL)
  // output: { url: string; finalUrl: string; status: number; contentType: string | null; content: string; truncated: boolean }

  • Tool name web_fetch; description says it fetches a public web page and returns its text, and that the content is untrusted.
  • GET only, no request body, no cookies, no caller-supplied headers; accept: text/html, text/plain, application/json, */*;q=0.5.
  • Redirects: redirect: 'manual', at most maxRedirects, only to http: / https:, each hop checked against allowedHosts / blockedHosts before DNS; the connection-time check is the pinned lookup.
  • Body: read the stream up to maxBytes, then cancel it and set truncated: true. Decode with the response charset (default UTF-8). HTML becomes text with a small built-in converter (drop script, style, noscript, template, svg; turn block elements and <br> into line breaks; keep link text with the URL in parentheses; decode the common named and numeric entities; collapse whitespace). JSON and text/* pass through. Other types (images, PDFs, binaries) return content: '' and a note in content saying the type is not supported.
  • A non-2xx status is not a thrown error: return it with the body text (as eve does), so the model can read a 404 page.
  • Errors (SsrfBlockedError, timeout, too many redirects, bad URL) become tool failures via toolFailure() from src/tools/built-in/toolFailure.ts, with messages that name the host but never the resolved private address list.
  • annotations: { readOnlyHint: true, openWorldHint: true }; no needsApproval by default.
  • Uses its own undici Agent({ connect: { lookup: pinnedLookup(...) } }) per tool instance, closed when the agent closes if a close hook exists, else left to GC (document).
  • A test-only transport option (not documented) so offline tests and the live-test replay can stand in for the network.
• Spec files: add 'web-fetch': webFetchTool to RESOLVABLE_BUILT_IN_TOOLS in src/spec/specToAgent.ts, and the Worker's built-in list stays as it is (web_fetch needs node:dns and undici; say so in docs).
• Docs: in docs/tools.md, a row for webFetchTool / createWebFetchTool(options) in the built-in tools table (line 123 area) and a short paragraph with the options (no new heading unless the page has one per tool; if it does, append at the end of that list); state that web_fetch is not available on Cloudflare Workers. Correct the httpTool row's claim to what the code does ("every resolved address is checked; the connection is not pinned, see web_fetch") so the docs stop overstating it.
• CHANGELOG entry; npm run docs:llms.

Out:

• Fixing http_request to use the pinned lookup (a separate ticket; this one adds the shared pinnedLookup it can adopt). Do not change http.ts behavior here.
• web_search: N1 covers provider-hosted search.
• PDF or image extraction, JavaScript rendering, robots.txt, caching.

Acceptance criteria

• src/security/privateAddress.test.ts: every listed range is private, including IPv4-mapped IPv6 (::ffff:127.0.0.1), NAT64 (64:ff9b::7f00:1), 6to4, 0.0.0.0, [::]; public addresses are not; pinnedLookup throws for a name resolving to any private address (stubbed resolver) and returns the checked address otherwise; allowPrivate exempts a matching host.
• src/security/credentialBroker.test.ts stays green using the shared list.
• src/tools/built-in/webFetch.test.ts against a local node:http server: HTML converted to text (script and style removed, entities decoded, links kept); JSON passed through; maxBytes truncation sets truncated; maxChars; a 404 returned with its body; 11 redirects fail with the default cap; a redirect to ftp: refused; allowedHosts and blockedHosts; the local server itself is refused by default (127.0.0.1) and allowed with allowPrivate: ['localhost']; DNS rebinding: with a stubbed resolver that answers a public address first and 127.0.0.1 second, the request never reaches the local server (the pinned lookup is the only resolution); timeout; non-http URL and URL with credentials refused.
• src/spec/specToAgent.test.ts: tools: ['web-fetch'] resolves.
• Docs and CHANGELOG as listed; snippets pass npm run docs:verify-snippets -- --skip-build; npm run docs:llms; every check in BRIEF-2.md "Verification" passes.

Live test

src/tools/built-in/webFetch.live.test.ts: an agent on recordReplay(() => resolveProvider('openrouter/openai/gpt-4o-mini'), { cassette: 'src/tools/built-in/__cassettes__/web-fetch.json', mode: process.env.LOUSHO_RECORD ? 'record' : 'replay' }) with createWebFetchTool(); prompt "Fetch https://example.com and tell me the page's heading in a few words." Assert the model called web_fetch with that URL and the answer mentions "Example Domain". In record mode the tool hits the real site; in replay mode the test passes the test-only transport returning a saved copy of the page, so CI touches neither the network nor the model. Record once with LOUSHO_RECORD=1 and the key loaded for that command only; grep the cassette for sk-or- and Authorization. Skip when the cassette is missing and no key is set. maxSteps: 3. Maximum spend: 0.05 USD.

Dependencies

None. Conflicts: src/security/credentialBroker.ts is also touched by M6 (the Docker CI job may change broker tests); src/spec/specToAgent.ts by M3.

Notes for the implementer

• undici's connect.lookup is called per connection (and per redirect target host), which is what makes the check and the dial use the same address; verify this with the rebinding test rather than assuming it.
• new URL() normalizes decimal, octal and hex IPv4 forms (see the comment at http.ts:120-126); IP-literal hosts skip DNS, so check them with isPrivateAddress directly before connecting.
• Report, do not fix, the http_request gap in the pull request description (link the follow-up issue you open), since BRIEF-2.md rule 6 keeps one ticket to one pull request.
• Keep the HTML-to-text converter small and dependency-free; it does not need to be perfect, only safe (never execute or follow anything) and readable.

---

Round 2 ticket N13a. Before starting, read the agent brief (worktree rules, verification list, live-test budget) and the plan. One ticket is one pull request; put Closes #<this issue> in it.

[LinuxDevil]

Starting this ticket (Claude Opus 5.5, 2026-10-02). Scope extended by the orchestrator: http_request will also adopt the shared pinned lookup in the same PR (security fix), with a rebinding regression test and corrected docs.

[aetherxeg-source]

One security/evidence boundary worth making explicit here is:

destination admitted != connection admitted

A hostname-policy decision is only authoritative if it is bound to the exact address actually consumed by the socket. A separate resolve/check followed by another resolver invocation creates two different execution objects even when both are described by the same hostname.

The pinned lookup direction closes that gap. I would also preserve the binding as explicit request evidence across redirects:

requested host -> resolved address set -> selected admitted address -> connection -> redirect hop

Each hop should create a new admission record; no prior host/address verdict should flow across the redirect boundary.

A compact invariant is:

checked address == dialed address

for every connection generation.

There is another subtle case around connection pooling: if an existing pooled connection is reused, the request did not perform a fresh dial even though a new policy evaluation may have occurred. The contract should state whether admission is bound to the connection's creation-time resolved address or whether policy changes require pool invalidation/new connections. allowPrivate changes are especially useful to test here.

A conformance matrix could therefore include DNS rebinding, redirect host changes, pooled-connection reuse, configuration changes between requests, and multi-address DNS where one result is private and another public. The evidence should make it impossible to report “host admitted” while the actual transport path is unknown or derived from a different resolution generation.

This separates hostname intent from the execution fact that matters for SSRF containment: which network endpoint actually received the request.