[N13a] web_fetch built-in with pinned-DNS SSRF checks; http_request no longer open to DNS rebinding

[LinuxDevil]

Closes #255

Adds the web_fetch built-in, and fixes a DNS-rebinding hole in http_request (scope extended by the orchestrator, because it is a security fix).

Security: http_request and DNS rebinding

The finding was real. isBlockedHost() resolved the name with dns.promises.lookup and checked the answer. Then fetch resolved the name again to connect. A name that answered a public address to the check and 127.0.0.1 to the connection got through. docs/tools.md, docs/deployment.md and the comment in src/deploy/runtime.worker.ts all called the check rebinding-safe.

Proof. The new test http.test.ts > "N13a: connects to the address it checked, so a rebinding name cannot reach a private address" stubs both resolver entry points with one shared answer sequence: public 203.0.113.10 first, 127.0.0.1 after that. I ran this test against the unchanged http.ts before the fix and it failed: promise resolved "'internal'" instead of rejecting, meaning the request reached the loopback server. After the fix the server gets no hit.

Fix.

• In-process path: connects through an undici Agent whose connect.lookup is the new shared pinnedLookup(). That lookup is the only resolution. It checks every address and gives the socket the address it checked. This holds for every redirect hop.
• Sandboxed path (sandboxExecute, which is what agents use): the host is resolved and checked in the agent's process. The sandboxed process then connects to that address (SandboxFetchRequest.pinnedAddress: node:http/https with a fixed lookup; Host and SNI keep the name). Two tests cover it:
  • A .invalid host whose check-time resolution is stubbed to 127.0.0.1 is only reachable through the pinned address.
  • A recording sandbox is given the checked address, and a private host never reaches sandbox.run().
• IP-literal hosts are checked before the request, as before.

Behavior changes (in the CHANGELOG under Security):

• The private list is now the shared, longer one.
• A name that does not resolve fails with the resolver's error.
• undici is now loaded on the first http_request request, not at import.
• New createHttpTool({ allowPrivate }) option. The existing local-server tests needed it, because they had relied on the old check-then-resolve-again gap.

Workers. There is no DNS hook on Workers, so neither tool exists in the Worker build. docs/deployment.md now says what a Worker can check (the URL, IP-literal hosts) and what it cannot guarantee (where a host name connects).

web_fetch (N13a)

• src/security/privateAddress.ts contains:
  • isPrivateAddress(): the broker's list plus 192.0.0.0/24, 198.18.0.0/15, 64:ff9b::/96 and 2002::/16, with IPv4-mapped unwrapping. Anything that is not an IP counts as private.
  • resolvePublicAddresses().
  • pinnedLookup().
  • SsrfBlockedError. Its message names the host only, never the addresses.
• The credential broker now uses the shared list.
• New files: src/tools/built-in/webFetch.ts (webFetchTool, createWebFetchTool, options as specified) and htmlToText.ts. htmlToText.ts is a linear-scan converter: no backtracking regex over the document, and nothing is executed or followed.
• 'web-fetch' is added to RESOLVABLE_BUILT_IN_TOOLS. The Worker tool list is unchanged.
• Docs: docs/tools.md (table rows and a paragraph with an options snippet; no new heading), docs/configuration.md (spec tools table row), docs/installation.md (undici note), docs/deployment.md (Worker paragraph). CHANGELOG has entries under Security and Added. npm run docs:llms was run.

One deviation from the ticket text: for unsupported content types the ticket said both content: '' and "a note in content". content carries the note ([web_fetch: content type image/png is not supported; ...]).

Acceptance criteria

• src/security/privateAddress.test.ts covers:
  • every range, IPv4-mapped, NAT64, 6to4, 0.0.0.0 and [::]
  • public addresses
  • pinnedLookup throwing, returning the checked address, and resolving exactly once
  • allowPrivate
• src/security/credentialBroker.test.ts is green on the shared list.
• src/tools/built-in/webFetch.test.ts runs against a local node:http server. It covers:
  • HTML to text, JSON, maxBytes, maxChars and a 404 with its body
  • 10 redirects succeed and the 11th fails
  • a redirect to ftp: is refused
  • allowedHosts / blockedHosts are applied before DNS
  • the local server is refused by default (name, 127.0.0.1, [::1]) and allowed with allowPrivate: ['localhost']
  • rebinding: public first, 127.0.0.1 second; the server is never hit and there is exactly one resolution
  • timeout, abort, and non-http, credential and invalid URLs
• src/spec/specToAgent.test.ts: tools: ['web-fetch'] resolves.
• Docs and CHANGELOG are done, snippets pass, and docs:llms was run. Every check below passed.
• Live test was not recorded. src/tools/built-in/webFetch.live.test.ts is written: record and replay, with the test-only transport replaying the saved page. It skips because there is no cassette. The record attempt was refused by OpenRouter with HTTP 402: the account's credits are used up (/credits: total_credits 10, total_usage ~10.20), even though the key's own counter shows usage 0 and limit_remaining 10. Recording needs the account topped up, then:

  LOUSHO_RECORD=1 OPENROUTER_API_KEY=... npx vitest run --config vitest.live.config.ts src/tools/built-in/webFetch.live.test.ts
  

Live test spend: before 0, after 0 (key usage). The one attempt was refused with 402 before any tokens were billed.

Verification

Run on the branch after merging origin/main (44538d5, M10a; the only conflict was CHANGELOG ### Added, where both sides were kept), head 2134c82:

tsc exit=0
lint exit=0                    (eslint src --max-warnings 0)
build exit=0
build-cla exit=0
test-types exit=0
snippets exit=0                (all 206 snippet(s) type-check against src/; 8 also run cleanly)
llms-check exit=0
coverage exit=0                Test Files 224 passed (224); Tests 3185 passed | 2 skipped; All files 90.5 | 91.62 | 93.67 | 90.5
fallow exit=0                  ✓ 0 above threshold · 4921 analyzed · maintainability 89.7 (good)
forge-typecheck exit=0
forge-typecheck-server exit=0
forge-test exit=0              13 files, 112 tests passed
forge-test-server exit=0       14 files, 122 tests passed
pack-smoke exit=0              [pack-smoke] all checks passed

The run before that merge (on a7b63c9) had one failure: src/execution/guardrails.test.ts, the E9 timeout test, with EPERM on removing its temp fixture. BRIEF-2 lists it as flaky under load. It passed alone (23/23) and in the full run above.

Rebinding test checked against the old code (finishing pass). I put origin/main's http.ts back and ran only the rebinding test: it failed with promise resolved "'internal'" instead of rejecting, so the request reached the loopback server. With the fix it passes. It now also asserts answers === 1: the pinned lookup is the only resolution.

openApiTools (merged in #283)

Its generated tools still use plain fetch with no private-address check, and docs/openapi-tools.md still says so. That stays true after this PR. Routing it through pinnedLookup is not a small change. It needs public API decisions:

• the default (it allows http://localhost on purpose, and base URLs are often internal);
• how it interacts with the user-supplied fetch option;
• whether it must stay runtime-neutral;
• whether to export a public pinned fetch helper.

Follow-up: #291.

For the docs site (G9)

• No new pages and no heading changes.
• New prose: docs/tools.md "Built-in tools" (table rows plus three paragraphs and a snippet), docs/configuration.md "Tools a spec can reference" (one row), docs/deployment.md cloudflare-worker tools bullet (rewritten), docs/installation.md (requirements line and the undici paragraph). The Arabic translations of these paragraphs need updating.

🤖 Generated with Claude Code