[browser][connectors] Expose the browser as a registry tool — let config/data agents declare browser_* (not only the Tier-0 workflow path)

[serge-ivo]

Situation

I was building a "Facebook Friends" agent (accept pending friend requests + add a few suggested friends — bounded and configurable). Working it through the platform boundary (docs/agent-platform-strategy.md: monorepo = platform only; agents = data or isolated code, never woven into core) surfaced a concrete, filable gap.

There are two ways to drive a browser on PAGS — one shipped, one planned:

1. Tier-0 workflow path — JobApplyWorkflow / INSURANCE_QUOTES, plus the planned generalization in epic [Epic] Generalize the job-apply browser engine into a general browser-task agent #69 ([browser] Extract the generic browser-task engine from the apply loop #70 extract engine → [browser] BrowserTaskWorkflow + objective-driven system prompt #71 BrowserTaskWorkflow → [browser] POST /v1/instances/:id/browse + browser.task runner type #72 POST /v1/instances/:id/browse + browser.task runner type → [browser] First-party Web Task catalog agent #73 first-party "Web Task" catalog agent). First-party, workflow-driven, surface/route-per-capability.

2. Tier-1 connector/tool registry — the framework from epic [Epic] Unified Connector + Tool framework — any system, any agent (runtime · MCP · API) #84 ([connectors] Single tool registry — one definition → runtime + MCP + API #85/[connectors] Connector framework — registry, auth, grants, scopes, tool bridge #86/[connectors] Generic tool-call API + MCP auto-generation #87/[connectors] Connector consent + write-scope safety + admin visibility #90). capabilities.tools is now a declared allowlist (agent-do-tools.ts:111, toolNamesFor — "the data-driven path that lets a third-party creator scope tools without a code change"); an agent gets exactly the catalog tools it names + BASE, dispatched through the AgentDO chat tool-loop. Connectors today: github, meta (whatsapp_send_message, instagram_send_dm), tmux.

The gap: there is no browser tool in the registry. A pure config/data (Tier-1) agent therefore cannot drive a browser — browser access exists only via the Tier-0 workflow path. Grep confirms no browser/computer/playwright connector in workers/api/src/lib/connectors/.

Why this blocks my case specifically:

• Facebook friend-confirming is a click-a-button-in-the-browser action, so it needs browser driving.
• It can't be an API connector instead: Meta's Graph API does not expose accepting friend requests or adding friends (hasn't been an API surface for ~a decade — the personal friend graph isn't exposed). Browser is the only path.
• So "Facebook Friends" cannot be built as data today, only as another Tier-0 special-case — which the strategy doc explicitly says not to add.

Need

Bridge the browser capability into the connector/tool registry (epic #84) as first-class registry tools, so browser driving becomes a declarable Tier-1 data tool — complementing, not replacing, the workflow path (#69).

Proposed tools (RegistryTool, connector: "browser", scope: "write", consent-gated via #90):

• browser_navigate({ url })
• browser_snapshot() → the ARIA [ref=eNN] tree the runner already produces
• browser_act({ action, ref, name, text?, ... }) → click/type/select/check/scroll/key — the runner's existing /browser/act primitives

Reuse the runner hands + the engine extracted in #70; no new per-agent code. A browser agent then becomes pure data:

capabilities.tools: ["browser_navigate","browser_snapshot","browser_act"]
+ goal prompt + guardrails + settingsSchema + board

…created via MCP/console, no commit. "Facebook Friends" (the flow #73 already names) becomes the first such config/data agent, and every future browser agent is likewise just a row.

Design question to resolve: workflow vs registry-tool

Two integration points for the same runner hands:

• Workflow ([browser] BrowserTaskWorkflow + objective-driven system prompt #71): durable, resumable past the 30s DO limit, captcha/stuck/needs_input handoff built in. Right for long, handoff-heavy tasks.
• Registry tool (this issue): driven by the AgentDO chat loop — the simplest "browser is just another tool." Right for bounded, interactive tasks, but inherits the chat loop's limits.

Recommend reconciling explicitly — both should sit on the engine extracted in #70. Likely: registry tool for bounded runs, escalating to / backed by the workflow for long ones.

Concrete sub-gaps the registry-tool path needs

• Handoff: captcha/stuck/needs_input handoff is currently bound to the workflow. The chat-loop path has none — a browser agent would die on the first Facebook checkpoint/2FA. Needs handoff, or must escalate to the workflow.
• Step budget: agent-think.ts maxToolRounds = 8 is too low for multi-step browser driving (each step is a snapshot→act pair). Needs headroom or a dedicated loop.
• Model: requires a tool-capable model (see Non-tool-capable default model silently disables ALL tools — a structured-data agent can't query its own collection #100 — non-tool default silently disables ALL tools).

Trust / safety (relationship to #74 / #75)

A browser tool runs on the subscriber's real Chrome profile, as them. Per #75, third-party creator use is gated on the trust model (domain/action allowlists, profile isolation, prompt-injection defenses, sensitive-action + mass-action gates, per-action audit). This issue should ship first-party / self-use only, behind an admin/experimental flag — exactly like #73 — and must not open browser tools to creators (that stays blocked on #75). The friend-add path is a "mass action" class → bounded caps + human approval by default. ToS/abuse caution: Facebook actively fights automation (checkpoints, throttling) — expect handoffs.

Related

Epic #69 (+ #70/#71/#72/#73) · epic #74 (+ #75/#76) · epic #84 (+ #85/#86/#87/#90) · #51 (Tier-1 declarative agent + open tool catalog + shared runtime) · #100 (tool-capable model) · #68 (focus wedge).

[serge-ivo]

Shipped the registry-tool path in 0f0fb9e.

Added a browser connector (workers/api/src/lib/connectors/browser.ts) that bridges the runner's real-Chrome hands into the Tier-1 tool registry — so a pure config/data agent can drive a browser by declaring capabilities.tools: ["browser_navigate","browser_snapshot","browser_act"], no per-agent code, no commit. Reached over the runner relay exactly like the tmux connector (auth:none, callRunner → the runner's existing /browser/snapshot + /browser/act).

• browser_navigate({url}) — write
• browser_snapshot() — read (the ARIA [ref=eNN] tree the runner already produces)
• browser_act({action, ref|role+name, text?, url?, key?, nth?, dy?}) — write; click/type/select/check/upload/navigate/scroll/key/wait

Safety (as required): EXPERIMENTAL, first-party/self-use ONLY — gated behind env BROWSER_TOOLS_ENABLED (fail-closed when unset; documented in wrangler.toml + the Env type). Writes additionally require the instance's browser write-consent (#90). Not opened to third-party creators — that stays blocked on the browser trust model (#75).

Tests: +7 in connectors/browser.test.ts (registration, scopes, flag gate fail-closed, consent gate fail-closed, no-runner error) + registry assertion updated. api + mcp typecheck clean; full api suite green (944).

Deliberately deferred (documented in the issue as sub-gaps, not built here)

• Handoff (captcha/stuck/needs_input): this bounded chat-loop path has none — a snapshot that detects a challenge reports it and stops rather than faking a solve. The handoff machinery lives on the durable workflow ([browser] BrowserTaskWorkflow + objective-driven system prompt #71); long/handoff-heavy tasks belong there.
• Step budget (agent-think.ts maxToolRounds = 8): untouched globally to avoid regressing other agents; a dedicated browser loop / higher budget is a follow-up.
• Model: needs a tool-capable model (Non-tool-capable default model silently disables ALL tools — a structured-data agent can't query its own collection #100, already resolved for the non-tool-default trap).

The 'Facebook Friends' agent the issue motivated can now be built as pure data (goal prompt + guardrails + capabilities.tools + the flag) once a first-party instance is granted browser write-consent.