Standardize discovery indexes for provider lookup and writeback context

[kjgbot]

Problem

Agents need stable lookup context for connected integrations even when historical provider records are not mounted. The immediate Slack case is channel writeback: an agent cannot write to a channel unless it can discover channel ids/names, but mounting /slack/channels/.../messages for history should remain optional.

Cloud PR AgentWorkforce/cloud#1931 adds a Slack-specific stopgap by materializing:

• /discovery/slack/channels/_index.json
• /discovery/slack/users/_index.json

and by triggering fetch-users + fetch-channels on Slack connection without requiring fetch-channel-history.

We should make this a provider-wide Relayfile adapter contract instead of an ad hoc Cloud-only Slack behavior.

Proposed contract

Every connected integration should expose a small /discovery/<provider>/... tree that is mounted independently of historical data sync. In addition to schemas/examples/action docs, discovery should include lookup indexes for provider entities agents need to reference when writing back.

Examples:

• Slack: /discovery/slack/channels/_index.json, /discovery/slack/users/_index.json
• GitHub: repo/org lookup indexes for PR/issue/review writeback
• Linear: teams, projects, states, users/assignees where supported
• Notion: databases/pages and any writable parent targets
• Jira/Confluence/GitLab/etc.: project/space/repo/resource lookup indexes needed by writeback schemas

Index rows should be compact, stable, and agent-friendly: ids, display names/titles, canonical read paths, writable resource/action paths, and any provider-specific fields required to disambiguate targets.

Acceptance criteria

• Define an adapter-level convention for discovery lookup indexes, including naming, row shape guidance, and where each provider documents its indexes in LAYOUT.md / .adapter.md.
• Update adapter resources/discovery docs so providers advertise lookup indexes alongside writeback schemas and examples.
• Add tests that fail when a provider has writable resources but no documented lookup/index surface for required target ids.
• Add or document the Cloud/materializer responsibility: generated/static adapter discovery assets come from relayfile-adapters; dynamic _index.json contents are produced from provider sync records and should be materialized even when historical mounts are disabled.
• Ensure this does not require mounting historical record trees. Discovery syncs for lookup entities should be independently triggerable on connect.

Notes

This likely spans two layers:

• relayfile-adapters: owns the public contract, provider docs, resource metadata, schemas/examples, and tests for what discovery should expose.
• cloud: consumes that contract, runs provider discovery syncs, and writes dynamic _index.json files into Relayfile workspaces.

Slack can be the reference implementation from Cloud #1931, but the long-term design should be generic enough that Pear/agents can rely on /discovery/<provider>/... for writeback context across all integrations.

[khaliqgant]

Implementation Plan

This is a protocol-level change affecting ~15-20 adapters. Filing a plan rather than implementing because:

1. Requires a per-adapter analysis of which target IDs are required for writeback
2. Needs a design decision on the enforcement mechanism
3. Slack is the only reference implementation; the contract needs to be formalized before replication

Reference implementation (Slack)

The complete pattern to replicate lives in:

• packages/slack/src/index-emitter.ts — buildSlackChannelsIndexFile, buildSlackUsersIndexFile
• packages/slack/src/path-mapper.ts — slackChannelsIndexPath(), slackUsersIndexPath()
• packages/slack/src/layout-prompt.ts — documents /discovery/slack/channels/_index.json row shapes
• packages/slack/discovery/slack/.adapter.md — lists discovery lookup index paths
• scripts/writeback-discovery-data.mjs lines 495-513 — readPaths array entries

Contract definition

Each adapter with writable resources that require target ID resolution must:

1. Declare /discovery/<provider>/<entity>/_index.json path helpers in path-mapper.ts
2. Export typed row shape interfaces (e.g. SlackChannelIndexRow)
3. Document the path and row schema in LAYOUT.md and .adapter.md
4. Register the path in scripts/writeback-discovery-data.mjs readPaths array
5. NOT emit the _index.json content itself (that's the cloud materializer's job at runtime)

Adapters prioritized by need

Tier 1 — Clear need for discovery lookup (no historical mount serves the use case):

• teams — channel/team ID lookup for message writeback (same pattern as Slack)
• hubspot — contact/deal/company ID lookup
• intercom — conversation/user ID lookup
• zendesk — ticket/user/organization ID lookup
• salesforce — record ID lookup by type
• pipedrive — deal/person/organization ID lookup

Tier 2 — Historical mount already provides indexes but /discovery/ variant may be useful:

• linear — teams, states, users already at /linear/teams/_index.json
• jira — projects at /jira/projects/_index.json
• github — repos at /github/repos/_index.json
• confluence — spaces at /confluence/spaces/_index.json

Tier 3 — No discovery index needed (runtime-keyed or no writeback target lookup):

• azure-blob, gcs, s3, dropbox, onedrive, redis, postgres, sharepoint, box

Enforcement mechanism (new)

Add a new test to scripts/verify-writeback-discovery.mjs that:

• Reads writeback-discovery-data.mjs per-adapter records
• For any adapter where resources has non-read-only resources with targetIdRequired: true (new field in the data), asserts that readPaths is non-empty
• Fails with a clear error listing the non-compliant adapters

Suggested implementation order

1. Formalize the contract in docs/writeback-spec-coverage.md (what "discovery lookup index" means, how to declare one)
2. Add teams as the second reference implementation (after Slack)
3. Add the enforcement test gate
4. Implement remaining Tier 1 adapters one PR each
5. Evaluate Tier 2 adapters (may just need .adapter.md cross-references to existing historical mount indexes)

Open questions

1. Should Tier 2 adapters add /discovery/<provider>/ variants OR just document that the historical mount indexes serve the same purpose?
2. The row schema for discovery indexes — should it be enforced by a JSON Schema in discovery/ alongside the writeback schemas?

[khaliqgant]

Implementation plan (scoped during issue-sprint triage)

This is a protocol-level contract touching every connected adapter, so it's too large for a single sprint PR. Recommend a phased rollout. Filing the design + phase breakdown.

Goal

Promote the Slack-specific stopgap (cloud#1931 materializing /discovery/slack/channels/_index.json + /discovery/slack/users/_index.json) into a provider-wide Relayfile adapter contract: every connected integration exposes a small, stable lookup index even when historical records aren't mounted.

Proposed contract

Add an optional emitter to the adapter interface, mirroring the existing emitAuxiliaryFiles / layoutPromptFile pattern:

// each adapter exports:
export function <provider>DiscoveryIndexFiles(ctx): AuxiliaryFile[]
// emitting /discovery/<provider>/<resource>/_index.json

Index row shape (provider-agnostic, lets agents resolve a writeback target without mounting history):

{ "id": "<stable-id>", "name": "<human-readable>", "type": "<resource>", "writebackRoot": "/<provider>/<resource>/<id>/..." }

The discovery/ directory already exists per-adapter for writeback schemas (.schema.json / .create.example.json); this extends it with runtime _index.json lookup data. Keep schema-only discovery and runtime-index discovery clearly separated (schemas are static/committed; indexes are materialized at sync).

Phasing

1. Phase 0 — contract + core helper. Add DiscoveryIndexFile type + a buildDiscoveryIndex() helper in @relayfile/adapter-core (sorting, row shape, path = /discovery/<provider>/<resource>/_index.json). Add a test:discovery-contracts check analogous to test:digest-contracts.
2. Phase 1 — Slack (reference). Reimplement the cloud#1931 stopgap as slackDiscoveryIndexFiles() in the adapter so cloud can drop its ad-hoc Slack code. This validates the contract end-to-end.
3. Phase 2 — writeback-first providers. github, linear, notion (resources agents actually write to). One PR per provider.
4. Phase 3 — remaining connected providers, driven by the matrix in docs/digest-layout-contract.md.

Why phased

A single PR touching ~15 adapters would be unreviewable and risks broad sync churn. Each phase is independently shippable and each adapter's index is additive (no reader breaks if a provider hasn't shipped its index yet — readers treat a missing /discovery/<p>/<r>/_index.json as "fall back to mounting").

Cross-repo note

Cloud's record-writer.ts consumes the single ADAPTERS registry, so once an adapter ships its discovery emitter cloud picks it up via the dep bump — no hand-copied Slack logic. cloud#1931's Slack-specific path can be removed after Phase 1 lands + publishes.

Triaged by the issue-sprint coordinator; phased because it's a cross-adapter protocol change, not a localized fix.