Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 40 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ add → ingest elements (store/chunk/embed) → process (optional, host)
tools / host ingest workflow → /api/tenants/:tenantId/memory/* → Memory plane → DocumentStore
Interchange auth + principal + grants

sidecar-bundle (deployed agent) → /api/workflow-memory/* → Memory plane → DocumentStore
hub credential + x-workflow-run-address (mountWorkflowMemory, no session)
```

Mount is intentionally small. The host already has `app`, grants, and
Expand Down Expand Up @@ -117,10 +121,42 @@ Returns an in-process `Memory` (`add`, `search`, `list`, `close`, plus the
optional retention writes) for host workers and ingestion modules that
already resolved identity.

**Agent tools live in this package** as thin HTTP clients
(`@corbits/memory/tools` / `interchange.tools`): `defineTool` factories that
`fetch` the mounted routes with install credentials. They do not import the
in-process plane. OpenAPI→MCP remains an optional host bridge.
**Capture** is the write path inside `add` (raw capture → chunks / edges /
embed on the default store). **Search** is hybrid retrieval on the same
plane, whether the caller arrived via tenant routes or the sidecar mount.

## Sidecar mount (`mountWorkflowMemory`)

Deployed agents do not install this package as a git sidecar. They carry
the factory at `@corbits/memory/sidecar-bundle`, which holds no client
code, no base URL, and no token: it resolves the host `hub` credential and
calls the run-scoped routes under `/api/workflow-memory/*`. That mount is
**parallel** to the tenant routes — two auth conventions stay on two
mounts so neither is harder to reason about.

```ts
import { mountWorkflowMemory } from "@corbits/memory";

mountWorkflowMemory(workflowMemoryApp, {
memory,
agentToken: { verify, resolveRun },
});
app.route("/api/workflow-memory", workflowMemoryApp);
```

Authorization on this mount **is the token itself**: the hub only mints an
agent token for a definition it already authorized, and every call is
confined to the verified run's tenant and principal. The mount runs **no
grant check of its own** and has no tenant override. A bearer minted for
one workbench cannot act on another's run (`verify` tenant must match
`resolveRun` tenant). Unrecognized bearer, unknown run address, and
cross-tenant mismatch all return the same **401**.

The sidecar factory (`src/sidecar-bundle.ts`) maps `memory_add` /
`memory_search` / `memory_list` / `memory_feed` onto those run-scoped
routes. Relative paths only — a mediated HTTP handle resolves them against
the origin it is pinned to. Capture and search still execute on the same
in-process `Memory` plane as the tenant routes.

## Provenance

Expand Down
86 changes: 68 additions & 18 deletions IMPLEMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,25 @@ and wire shapes. For the "why standalone" / boundaries story, read

```
src/
index.ts # createMemory / registerMemoryRoutes + distiller re-exports
index.ts # createMemory / registerMemoryRoutes / mountWorkflowMemory + distiller re-exports

mount-config.ts # MemoryConfig + loadMemoryConfig() — the mount config
config.ts # EngineConfig — the core vector-plane config (db + embed + rerank)
memory.ts # createMemory — add/search/list against store or pgvector
grant-tags.ts # resolveAccessTags + canAccessDocument (host grants)
workflow-mount.ts # mountWorkflowMemory — run-scoped /api/workflow-memory/*
sidecar-bundle.ts # @corbits/memory/sidecar-bundle factory (no client, no token)
tools.ts # MEMORY_TOOL_DEFINITIONS — sidecar binds names to run-scoped routes
http-client.ts # host-side HTTP client for tenant routes (imperative distill tick)

log.ts # getLogger(["memory"]) from @intx/log
migrations.ts # runMemoryMigrations(url)
ports/ # DocumentStore / SourceProvider + fakes
routes/ # the mounted routes
routes/ # the mounted tenant routes
mount.ts # registerMemoryRoutes (HTTP)

deps.ts # RouteDeps, caller(c) (context identity), grantGuard
add.ts, search.ts, list.ts, feed.ts
tools/ # Interchange defineTool factories (HTTP clients)
add.ts, search.ts, list.ts, feed.ts, client.ts, install.ts
distiller/ # Resident distiller (CL-5869) — workflow + tick helpers
index.ts # createResidentDistiller, runDistillTick, buildDistilledClaim
workflow.ts # defineWorkflow + defineAgent with memory tools
Expand Down Expand Up @@ -640,20 +642,68 @@ surface, or a migrating host silently loses them.

`registerMemoryRoutes` and `createMemory({ app })` register these seven HTTP
routes (add, search, list, feed, forget, purge, retention-class).
Agent tools ship in this package as Interchange `defineTool` factories
(`@corbits/memory/tools` / `interchange.tools`): thin HTTP clients that call the
mounted routes with install env (`memoryBaseUrl`, `memoryTenantId`,
`memoryAuthToken`). They do not import the plane. Host checklist: agent principal
needs `memory:add` and/or `memory:search` grants; Bearer token only (no session
cookie path); tool results are JSON strings; pass `AbortSignal` if you need hang
protection — the client has no default timeout. OpenAPI→MCP remains an optional
host bridge. The plane surface is `add` / `search` / `list` / `close`, plus
optional transform methods when backed by the engine DocumentStore
(`createTransformConfig`, `listTransformConfigs`, `runTransform`,
`promoteGeneration`, `demoteGeneration`) and optional retention methods
(`tombstoneDocument`, `hardDeleteDocument`, `setRetentionClass`,
`sweepEphemeral`, `deprecateVersion`) — see docs/RETENTION.md. Inference stays
on the host.
**Capture** (`services/capture.ts`) is the write path inside `add`.
**Search** (`services/search.ts`) is hybrid retrieval. Deployed agents do
not call these tenant routes with a git-installed client; they use the
sidecar mount below. The plane surface is `add` / `search` / `list` /
`close`, plus optional transform methods when backed by the engine
DocumentStore (`createTransformConfig`, `listTransformConfigs`,
`runTransform`, `promoteGeneration`, `demoteGeneration`) and optional
retention methods (`tombstoneDocument`, `hardDeleteDocument`,
`setRetentionClass`, `sweepEphemeral`, `deprecateVersion`) — see
docs/RETENTION.md. Inference stays on the host. Host workers that already
have HTTP to the tenant tree can use `createMemoryHttpClient`
(`src/http-client.ts`) — that is a host client, not the agent sidecar.

### Run-scoped sidecar (`mountWorkflowMemory`)

`src/workflow-mount.ts`, re-exported from the barrel. Parallel to the
tenant tree — do not fold agent-bearer auth into `registerMemoryRoutes`.
`package.json` exports `@corbits/memory/sidecar-bundle` →
`src/sidecar-bundle.ts`.

```ts
import { mountWorkflowMemory } from "@corbits/memory";

mountWorkflowMemory(workflowMemoryApp, {
memory,
agentToken: { verify, resolveRun },
});
app.route("/api/workflow-memory", workflowMemoryApp);
```

| Constant | Value |
| --- | --- |
| `WORKFLOW_MEMORY_BASE_PATH` | `/api/workflow-memory` |
| `HUB_CREDENTIAL_HANDLE` | `hub` |
| `SIDECAR_BUNDLE_ID` | `@corbits/memory/sidecar-bundle` |

**Auth.** Every route sits behind middleware: `agentToken.verify(c)` reads
the presented `Authorization`; `agentToken.resolveRun` looks up
`x-workflow-run-address`. Same **401** body whether the bearer is
unrecognized, the address names no run, or the run's tenant is not the
token's tenant. No `requireGrant`. No tenant override. Scope on context:
`workflowRunScope` `{ tenantId, principalId, runId }`.

**Wire (relative to the mount; sidecar never names a host).** Responses
are `{ data: … }` on success. Sidecar `requires`: `capabilities`,
`address` (run address). Tool results are JSON strings.

| Tool name | Method + path | Notes |
| --- | --- | --- |
| `memory_add` | `POST /add` | Body same as tenant add. Forces `share.tenant = true` (team share; explicit share only widens). Capture path is `memory.add` → `captureDocument`. |
| `memory_search` | `POST /search` | Body same as tenant search; `visibleTags` = `[tenantTag(tenantId)]`. |
| `memory_list` | `GET /list?limit=` | Same grant-tag visibility as search (`visibleTags` team tag). |
| `memory_feed` | `GET /feed?after=&limit=&exclude_generator=` | **501** if `memory.feed` is undefined on this plane. |

Unknown tool name → tool error (`isError: true`). Non-HTTP hub credential
kind → error. `callMemoryRoute` sends `x-workflow-run-address` and
optional JSON body; mediated `credential.fetch` injects the bearer and
pins origin.

`callerResolver` on the **tenant** routes is a different seam (machine
caller through the same grant path). Do not treat it as a replacement for
`mountWorkflowMemory`.

### Share materialization (CL-5873)

Expand Down
44 changes: 30 additions & 14 deletions PRODUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@
Memory for Interchange hubs: durable documents, hybrid search, recent list.

**You mount it on the hub (~5 lines). That exposes protected routes. Agents
and ingestion modules call those routes.** Workbench and coding agents are
clients — not owners of auth. Inference stays host-injected.
and ingestion modules call those routes.** Capture (`add`) writes; search
retrieves. Workbench and coding agents are clients — not owners of auth.
Inference stays host-injected. Deployed agents do **not** git-install this
package: they carry `@corbits/memory/sidecar-bundle` and hit the parallel
`mountWorkflowMemory` routes.

## Default pipeline (locked)

Expand Down Expand Up @@ -34,10 +37,11 @@ never creates one; it mounts onto yours.
| Surface | Role |
| --- | --- |
| `createMemory({ app, … })` | Register `/api/tenants/:tenantId/memory/*` + return the plane |
| `mountWorkflowMemory(app, { memory, agentToken })` | Parallel run-scoped `/api/workflow-memory/*` for deployed agents |
| `loadMemoryConfig()` | Config from env |
| `runMemoryMigrations(url)` | Apply pgvector schema |
| `registerMemoryRoutes` | Low-level HTTP only (optional) |
| `@corbits/memory/tools` | Interchange tools (`memory_add` / `search` / `list` / `feed`) |
| `@corbits/memory/sidecar-bundle` | Deployed-agent factory — no client code, no base URL, no token |
| `@corbits/memory/distiller` | Optional process helpers: `runDistillTick`, `createResidentDistiller` |

### Verbs
Expand All @@ -55,9 +59,11 @@ share-grant materialization. Process helpers:
`createResidentDistiller` / `runDistillTick` (`docs/DISTILLER.md`) — host
injects inference; not the default ingest path.

Identity is always **`principalId` + `tenantId`** on the plane. HTTP routes
never take body identity — they read `c.get("principal")` from Interchange
context.
Identity is always **`principalId` + `tenantId`** on the plane. Tenant HTTP
routes never take body identity — they read `c.get("principal")` from
Interchange context. Run-scoped sidecar routes read the verified workflow
run (`agentToken.verify` + `x-workflow-run-address`); the sidecar factory
never names a host or carries a token.

### How it is used

Expand All @@ -66,29 +72,37 @@ Agent / host ingest workflow
│ tool call or host worker
│ → POST|GET /api/tenants/:tenantId/memory/*
│ authenticated by Interchange (session | API key | MCP OAuth)
Deployed agent (sidecar-bundle)
│ hub credential + run address
│ → POST|GET /api/workflow-memory/* (mountWorkflowMemory)
┌──────────────────────────────────────────────┐
│ Host Interchange createApp │
│ principal + tenant on context │
│ + createMemory({ app, grantStore, … }) │
│ grants: memory:add | memory:search │
│ documentStore: pgvector | host | fake │
│ + mountWorkflowMemory(app, { memory, … }) │
│ │ in-process │
│ ▼ │
│ Memory plane: add / search / list
│ Memory plane: add (capture) / search / list │
│ → DocumentStore (sole durable backend) │
└──────────────────────────────────────────────┘
```

1. **Mount** — host passes `app` + the same grant store it already uses.
2. **Tools** — install `@corbits/memory/tools` (`defineTool` factories) on a
workflow with env credentials (`memoryBaseUrl`, `memoryTenantId`,
`memoryAuthToken`). Tools HTTP-call the mounted routes; identity is the
hub-authenticated principal. OpenAPI→MCP remains an optional host bridge.
2. **Sidecar (deployed agents)** — agents carry
`@corbits/memory/sidecar-bundle`. It holds no client code, no base URL,
and no token: it resolves the host `hub` credential and calls
`/api/workflow-memory/*`. Host wires `mountWorkflowMemory` in parallel
with the tenant routes (see README How it works). Not the primary
install — that is still `createMemory` + `loadMemoryConfig`.
3. **Ingestion** — preferred: one host workflow (or module) does
**add → ingest elements → process**. Mechanical ingest is inside `add` on
the default store; process (claims / links) is host-injected inference in
the same pipeline when you want a company brain.
**add → ingest elements (capture) → process**. Mechanical capture is
inside `add` on the default store; process (claims / links) is
host-injected inference in the same pipeline when you want a company
brain.

### Ports

Expand All @@ -106,6 +120,8 @@ stores, Linear tools. Core never imports vendor SDKs.
- No answer/generation endpoint — host owns inference.
- Workbench is a client, not required.
- Core does not run the ingest workflow process — the host does.
- Deployed agents do not git-install this package as a sidecar and do not
receive a memory base URL or token — that is the sidecar-bundle contract.

**Default durable store:** Postgres via `DATABASE_URL`, tables under the
**`memory`** schema. When
Expand Down
Loading
Loading