From 20e8471deaf288a8065a5856e517931cc1b2538a Mon Sep 17 00:00:00 2001 From: Sawyer Cutler Date: Thu, 24 Sep 2026 23:49:04 -0700 Subject: [PATCH 1/2] docs(distiller): destructure createResidentDistiller in its example The function returns { workflow, agent, generatorAgentId }, not the workflow alone. --- src/distiller/workflow.ts | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/distiller/workflow.ts b/src/distiller/workflow.ts index ff58836..5137027 100644 --- a/src/distiller/workflow.ts +++ b/src/distiller/workflow.ts @@ -4,13 +4,13 @@ * ```ts * import { createResidentDistiller } from "@corbits/memory/distiller"; * - * const workflow = createResidentDistiller({ + * const { workflow, agent } = createResidentDistiller({ * mailTo: "resident-distiller@tenant.example.com", * inference: { sources: [{ provider: "openai", model: "gpt-4.1-mini" }] }, * }); - * // deploy with host workflow-deploy; the host binds the `hub` credential - * // handle the sidecar bundle resolves at run time, and ticks the workflow - * // by mailing `mailTo` (e.g. a @corbits/cron schedule) — see the README. + * // deploy `workflow` and `agent` with host workflow-deploy; the host binds + * // the `hub` credential handle the sidecar bundle resolves at run time, and + * // ticks the workflow by mailing `mailTo`. * ``` */ import { From e2ece628baba81524daae64b1a834b8ba4867585 Mon Sep 17 00:00:00 2001 From: Sawyer Cutler Date: Thu, 24 Sep 2026 23:49:04 -0700 Subject: [PATCH 2/2] docs(readme): rewrite around what the package is The opening says what the package is and when to use it. Environment variables move from a quickstart paragraph into a Reference table. Undefined internal terms are replaced with plain wording, the resident distiller is introduced once, the cron wiring becomes one line, and the Development section moves to CONTRIBUTING. --- CONTRIBUTING.md | 15 +- README.md | 319 +++++++++++++++++++------------------- src/distiller/workflow.ts | 6 +- 3 files changed, 167 insertions(+), 173 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0544612..e510af5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,15 +8,16 @@ Thanks for considering a contribution to Corbits Memory. git clone https://github.com/corbitsdev/corbits-memory.git cd corbits-memory docker compose up -d # pgvector Postgres on localhost:5434 -cp .env.example .env # edit as needed — see README.md's quickstart +cp .env.example .env # edit as needed — see README.md's Reference bun install -bun run db:setup # applies migrations/*.sql, idempotent -bun run dev # bun --watch src/server.ts +bun run db:setup # applies migrations/*.sql, idempotent +bun run build # compiles src/ to dist/, which the package publishes ``` -Requires Bun 1.2+. See `README.md` for the full quickstart including a -zero-cost local embedding endpoint (Ollama), and `IMPLEMENTATION.md` for how -the pieces fit together. +Requires Bun 1.2+. `compose.yml` also runs a local Ollama for embeddings +(`docker compose exec ollama ollama pull nomic-embed-text`). Unit tests use +the in-repo `createFakeDocumentStore`/`createFakeSourceProvider` and need no +Postgres. See `IMPLEMENTATION.md` for how the pieces fit together. ## Running the tests @@ -48,7 +49,7 @@ numbered file, because a guarded `ADD CONSTRAINT` keeps the old definition. - Branch off `main`; open PRs against `main`. - Keep PRs scoped to one logical change — a mix of an unrelated refactor and a feature makes review slower, not faster. -- Describe *why* the change is needed in the PR description, not just what +- Describe _why_ the change is needed in the PR description, not just what changed; link any relevant issue. - Make sure `bun run typecheck && bun run test` pass before requesting review. diff --git a/README.md b/README.md index c5ef414..bcbd25b 100644 --- a/README.md +++ b/README.md @@ -1,214 +1,207 @@ # @corbits/memory -Memory for Interchange hubs: **add**, **search**, **list**. Mount it on the -hub; routes land under `/api/tenants/:tenantId/memory/*` so the hub's -existing tenant middleware supplies principal + tenant. Host workers call -the same plane in-process. Inference stays host-owned — this package does -not ship an answer endpoint. +[![npm](https://img.shields.io/npm/v/@corbits/memory.svg)](https://www.npmjs.com/package/@corbits/memory) [![License: LGPL-2.1](https://img.shields.io/badge/license-LGPL--2.1-green.svg)](https://github.com/corbitsdev/corbits-memory/blob/main/LICENSE) -## Runtime support +Hybrid semantic and full-text document memory in Postgres with pgvector, with optional embedding and rerank endpoints. A Corbits hub module: it mounts Hono routes on `@intx/hub-api` that check Interchange grants (permissions a principal, a user or agent account, holds on a resource), and ships agent tools for the sidecar, the Interchange agent runtime. -The package ships compiled JavaScript and type declarations in `dist/`, so it -runs on Node 22+ or Bun. +## Why @corbits/memory? -Peer dependencies (the stack an Interchange hub already has): `@intx/agent`, -`@intx/authz`, `@intx/hub-api`, `@intx/log`, `@intx/types`, `@intx/workflow`, -`drizzle-orm`, `hono`, `hono-openapi`, `postgres`. +1. **One store for people and agents.** Users add documents through the tenant routes, and deployed agents read and write the same store through run-scoped routes and a sidecar tool pack. +2. **Access follows grants.** Each document carries access tags. A caller sees a document only when a grant covers one of its tags, and the creator always sees their own. +3. **Search degrades instead of failing.** With no embedding endpoint, or when one is down, search falls back to Postgres full-text search and says so in a `degraded` field. +4. **Migrations that replay safely.** Every SQL file is idempotent, so running the migrations again is a no-op. -## Quickstart +It does not generate answers: inference stays with the host, which calls `search` and passes the results to its own model. + +## Install ```bash -npm add @corbits/memory -pnpm add @corbits/memory -yarn add @corbits/memory -bun add @corbits/memory +bun add @corbits/memory \ + @intx/agent @intx/authz @intx/db @intx/hub-api @intx/log @intx/types @intx/workflow \ + drizzle-orm hono hono-openapi postgres ``` -Build the plane, then mount its routes on your hub's `app` with the -`grantStore` and `conditionRegistry` you already pass to -`createRequireGrant`/`createApp`: +Postgres needs the pgvector extension. + +## Quickstart + +With `DATABASE_URL` pointing at a hub database that has run `runMemoryMigrations` (see [Using with Interchange](#using-with-interchange)): ```ts -import { Hono } from "hono"; -import { createRequireGrant, type TenantEnv } from "@intx/hub-api"; -import type { ConditionRegistry, GrantStore } from "@intx/authz"; -import { - createMemory, - createMemoryRoutes, - loadMemoryConfig, - type Memory, -} from "@corbits/memory"; +import { createMemory, loadMemoryConfig } from "@corbits/memory"; -export function installMemory( - app: Hono, - grantStore: GrantStore, - conditionRegistry: ConditionRegistry, -): Memory { - const memory = createMemory({ - config: loadMemoryConfig(), // DATABASE_URL + embed env — see below - grantStore, - conditionRegistry, - }); - app.route( - "/api/tenants/:tenantId/memory", - createMemoryRoutes({ - memory, - requireGrant: createRequireGrant({ grantStore, conditionRegistry }), - }), - ); - return memory; -} +const memory = createMemory({ config: loadMemoryConfig() }); + +await memory.add({ + tenantId: "acme", + principalId: "alice", + content: { title: "Deploys", text: "Staging deploys run from main." }, +}); +console.log( + await memory.search({ + tenantId: "acme", + principalId: "alice", + query: "staging", + }), +); +await memory.close(); ``` -Mount `installMemory` below the middleware that sets `principal`/`tenant` -(a real hub's `createResolveTenant` on `/api/tenants/:tenantId/*` already -does). Identity comes from `c.get("principal")` — request bodies never -carry tenant or principal. Missing principal → 401, missing grant → 403. +`acme` and `alice` must be a tenant and principal in the hub. The search returns the document just added. -Apply migrations before serving traffic, with the same `DBConfig` and -`schema` your hub passes to Interchange `runMigrations`. Memory's tables land -in their own `memory` schema, with foreign keys into that schema's `tenant` -and `principal` tables: +## Where it fits -```ts -import { runMemoryMigrations } from "@corbits/memory/migrations"; +[Interchange](https://github.com/faremeter/interchange) runs AI agents as principals: accounts with their own identity, permissions and credentials. Its hub is the multi-tenant control plane that holds tenants, principals and grants (permissions a principal holds on a resource); its sidecar is the agent runtime. -await runMemoryMigrations(dbConfig, { schema: "public", ftsLanguage: "english" }); -``` +- **Runs in:** the hub, as routes on its Hono app and tables in its Postgres (`memory` schema). +- **Plugs into:** [`@intx/hub-api`](https://github.com/faremeter/interchange/tree/main/packages/hub-api) routes and grants, [`@intx/db`](https://github.com/faremeter/interchange/tree/main/packages/db) (its `DBConfig`, and its `tenant` and `principal` tables as foreign-key targets), [`@intx/agent`](https://github.com/faremeter/interchange/tree/main/packages/agent) tools on the sidecar, and [`@intx/workflow`](https://github.com/faremeter/interchange/tree/main/packages/workflow) for the resident distiller. +- **Pairs with:** [`@corbits/embedding`](https://github.com/corbitsdev/corbits-embedding) and [`@corbits/reranking`](https://github.com/corbitsdev/corbits-reranking) endpoints, [`@corbits/agent-token`](https://github.com/corbitsdev/corbits-agent-token) for agent bearer tokens, and [`@corbits/cron`](https://github.com/corbitsdev/corbits-cron) to tick the distiller. -`loadMemoryConfig()` reads `DATABASE_URL` (required — tables live in a -`memory` Postgres schema, pgvector-capable) plus the embed pair: -`EMBED_BASE_URL`/`EMBED_MODEL` (OpenAI-compatible by default; `EMBED_API_KEY` -optional, `EMBED_API_STYLE`/`EMBED_TIMEOUT_MS` to override). Set both to -enable dense retrieval, or leave both unset to run lexical-only — full-text -search with no embed endpoint, a fully-supported mode rather than a degraded -one. Setting exactly one throws at load time. There is no credential-based -fallback: the embed pair is deployment config, resolved once per process -from the environment, the same for every tenant it serves. `RERANK_BASE_URL`/ -`RERANK_MODEL`/`RERANK_API_KEY` are the equivalent optional pair for -reranking. See `.env.example` in this repo for the full list. +## Reference -### Deployed agents (run-scoped routes) +### `createMemory(options)` -A deployed agent authenticates with its own sidecar bearer token, not a -browser session, so it gets a second mount, scoped to the run rather than to -a tenant-session request: +| Option | Type | What the host provides | +| ------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------- | +| `config` | `MemoryConfig` | Database and endpoint settings, usually from `loadMemoryConfig()`. Required unless `documentStore` is set. | +| `documentStore` | `DocumentStore` | Optional. A custom storage backend in place of the Postgres engine. | +| `grantStore` | `GrantStore` | Optional. The hub's grant store. Without it, search and list return only the caller's own documents. | +| `conditionRegistry` | `ConditionRegistry` | Optional. The hub's condition registry for conditional grants. | +| `textExtractor` | `TextExtractor` | Optional. Turns `add({ file })` bytes into text; without it, only `add({ content })` is served. | -```ts -import { Hono } from "hono"; -import type { TenantEnv } from "@intx/hub-api"; -import { - mountWorkflowMemory, - type Memory, - type WorkflowMemoryEnv, -} from "@corbits/memory"; +The returned `Memory` has `add`, `search` and `list`, which take the caller's `tenantId` and `principalId`; `feed` on the Postgres engine; `capabilities`, whose `embeddingsConfigured` says whether dense retrieval is on; and `close`. -export function installWorkflowMemory( - app: Hono, - memory: Memory, - agentToken: { - verify: ( - ctx: unknown, - ) => Promise<{ tenantId: string; definitionId: string } | undefined>; - resolveRun: (runAddress: string) => Promise<{ - tenantId: string; - principalId: string; - runId: string; - } | null>; - }, -): void { - const workflowMemoryApi = new Hono(); - mountWorkflowMemory(workflowMemoryApi, { memory, agentToken }); - app.route("/api/workflow-memory", workflowMemoryApi); -} -``` +Other root exports: `MemoryError` and `RerankConfigError`, the `SEARCH_LIMIT_*` and `LIST_LIMIT_*` bounds, `MEMORY_GRANT_REQUIREMENTS` and `capabilityIdsForSurface`, `MEMORY_TOOL_DEFINITIONS`, `createMemoryHttpClient` for calling the routes over HTTP, and the `DocumentStore` port types. `loadMemoryConfig` is also at `@corbits/memory/config`. -`memory` here is the same plane `installMemory` built — one engine, two -mounts. The tool definitions a deployed agent calls against this mount ship -at `@corbits/memory/sidecar-bundle`. +### `createMemoryRoutes(deps)` -## How it works +Returns a `Hono` sub-app. Mount it at `/api/tenants/:tenantId/memory`, below the middleware that sets `principal` and `tenant` on the context (Interchange's `createResolveTenant` does). Identity always comes from the context, never from the request body. -`createMemory` builds the plane. `createMemoryRoutes` returns its HTTP routes -as a Hono sub-app, each guarded by `requireGrant("memory", …)`. -`loadMemoryConfig` lives on the barrel and at `@corbits/memory/config`. +| Route | Grant | Does | +| ------------------------------------------- | --------------- | ---------------------------------------------------------------- | +| `POST /add` | `memory:add` | Adds a document, or a new version of one. | +| `POST /search` | `memory:search` | Hybrid search over the documents the caller can see. | +| `GET /list` | `memory:search` | Recent documents the caller can see. | +| `GET /feed` | `memory:search` | New versions after a `feed_seq` cursor, for consumers. | +| `POST /documents/:documentId/forget` | `memory:forget` | Drops a document from search and redacts its text. Creator only. | +| `POST /documents/:documentId/purge` | `memory:purge` | Deletes a document and its versions. Creator only. | +| `POST /versions/:versionId/retention-class` | `memory:forget` | Sets a version's retention class. Creator only. | -Capability grants (`memory:add` / `memory:search` / `memory:forget` / -`memory:purge`) gate the routes. Per-document visibility is Interchange -grant tags on the row (`access_tags`); the creator always sees their own -docs. Details: [`docs/AUTHZ-DOCUMENT-ACCESS.md`](docs/AUTHZ-DOCUMENT-ACCESS.md). +A missing principal answers `401` and a missing grant answers `403`. `callerResolver` is optional: it resolves a non-session caller to a `{ tenantId, principalId }` before the same grant checks run, and a malformed result answers `500`. -The resident distiller (`createResidentDistiller` / `runDistillTick`) is at -`@corbits/memory/distiller`. +### `runMemoryMigrations(config, { schema, ftsLanguage })` -### Resident distiller +Takes the same `DBConfig` and `schema` as Interchange's `runMigrations`. `schema` holds the host's `tenant` and `principal` tables; this package's tables always go in `memory`. `ftsLanguage` is fixed into the full-text index and must match `FTS_LANGUAGE` at runtime. -`createResidentDistiller` returns a mail-triggered workflow + agent, preloaded -with the memory sidecar tools. It never ships a scheduler — the host owns -ticking: +Each run replays every file in one transaction behind a Postgres advisory lock, so every replica can call it at boot: concurrent runs wait their turn. A run also waits behind long queries on memory's tables rather than timing out. -```ts -import { createResidentDistiller } from "@corbits/memory/distiller"; +### `loadMemoryConfig()` -const { workflow, agent } = createResidentDistiller({ - mailTo: "resident-distiller@tenant.example.com", - inference: { sources: [{ provider: "openai", model: "gpt-4.1-mini" }] }, -}); -// deploy `workflow` (and `agent`) with the host's workflow-deploy; the run -// fires on the first inbound mail at `mailTo` and stays live for the host -// to keep addressing. -``` +Reads the environment. `DATABASE_URL` must point at the same database as the `DBConfig` passed to `runMemoryMigrations`. Set `EMBED_BASE_URL` and `EMBED_MODEL` together for dense retrieval, or leave both unset for full-text search only. + +| Variable | Default | Meaning | +| ---------------------- | ---------------------- | --------------------------------------------------- | +| `DATABASE_URL` | required | Postgres with pgvector. | +| `DB_POOL_MAX` | `8` | Connection pool size. | +| `FTS_LANGUAGE` | `english` | Postgres text search configuration. | +| `EMBED_BASE_URL` | none | Embedding endpoint. | +| `EMBED_MODEL` | none | Embedding model name. | +| `EMBED_API_STYLE` | `openai` | `openai`, `tei` or `ollama`. | +| `EMBED_API_KEY` | none | Bearer token for the embedding endpoint. | +| `EMBED_TIMEOUT_MS` | `10000` | Embedding request timeout. | +| `RERANK_BASE_URL` | none | Cross-encoder rerank endpoint. | +| `RERANK_MODEL` | none | Rerank model name. | +| `RERANK_API_KEY` | none | Bearer token for the rerank endpoint. | +| `RERANK_MAX_DOC_CHARS` | derived from the model | Per-document character budget sent to the reranker. | +| `RERANK_TIMEOUT_MS` | `10000` | Rerank request timeout. | + +### `mountWorkflowMemory(app, { memory, agentToken })` + +Registers `/add`, `/search`, `/list` and `/feed` for deployed agents. A request carries the agent's bearer token and an `x-workflow-run-address` header; `agentToken.verify` checks the token and `agentToken.resolveRun` maps the address to the run's tenant and principal. Every call is confined to that run. + +### `@corbits/memory/sidecar-bundle` + +`memory` is an `@intx/agent` tool pack: `memory_add`, `memory_search`, `memory_list` and `memory_feed`. They call the run-scoped routes at `/api/workflow-memory` through the agent's `hub` credential. + +### `@corbits/memory/distiller` -Ticking is a host concern, not this package's: point a scheduler at -`mailTo` on whatever cadence you want. If the host already runs -`@corbits/cron`, wire a schedule whose delivery mails `mailTo` — the same -pattern a cron-ticked workflow uses everywhere on Interchange: the ticker -polls for due rows and hands each one to the host's mail transport, so a -mail-triggered workflow is ticked by addressing it directly. No import of -`@corbits/cron` is required here; see its own README for `mountCron` / -`createCronTicker` wiring. +`createResidentDistiller({ mailTo, inference })` returns a workflow and an agent that turn new versions from the feed into distilled claims. Each mail to `mailTo` runs one pass; the host decides when to send it. -### Lower-level: in-process calls, no HTTP +## Using with Interchange -A host worker can skip `createMemoryRoutes` and call `add`/`search` on the -plane directly (the resident distiller does this): +Run the migrations after Interchange's, mount the tenant and run-scoped routes on the same `Memory`, grant `memory:add` and `memory:search` to principals that use it (and `memory:forget` and `memory:purge` to creators who may retract their documents), and give agents the tool pack. ```ts -import { createMemory, loadMemoryConfig } from "@corbits/memory"; +import { timeWindowEvaluator } from "@intx/authz"; +import { createDB, createGrantStore, runMigrations } from "@intx/db"; +import { createRequireGrant } from "@intx/hub-api"; +import { + createMemory, + createMemoryRoutes, + loadMemoryConfig, +} from "@corbits/memory"; +import { runMemoryMigrations } from "@corbits/memory/migrations"; -const memory = createMemory({ config: loadMemoryConfig() }); +const dbConfig = { + host: "localhost", + port: 5432, + user: "postgres", + password: "postgres", + database: "interchange", +}; + +await runMigrations(dbConfig, { schema: "public" }); +await runMemoryMigrations(dbConfig, { + schema: "public", + ftsLanguage: "english", +}); -await memory.add({ - tenantId: "acme", - principalId: "alice", - content: { title: "Deploy notes", text: "Staging deploys run from main." }, +const { db } = createDB(dbConfig); +const grantStore = createGrantStore(db); +const conditionRegistry = { time_window: timeWindowEvaluator }; +const memory = createMemory({ + config: loadMemoryConfig(), + grantStore, + conditionRegistry, }); -const { items } = await memory.search({ - tenantId: "acme", - principalId: "alice", - query: "staging", +export const memoryRoutes = createMemoryRoutes({ + memory, + requireGrant: createRequireGrant({ grantStore, conditionRegistry }), }); ``` -Without `grantStore`, search/list fall back to creator-only visibility — a -safe default for a standalone caller, but not the shared-document behavior a -real tenant gets through `installMemory` above. +Mount `memoryRoutes` on the hub app at `/api/tenants/:tenantId/memory`, behind the hub's auth and tenant middleware. -## Development +For agents, mount `mountWorkflowMemory(new Hono(), { memory, agentToken })` at `/api/workflow-memory`. `agentToken` is the host's `{ verify, resolveRun }`: `verify` checks the agent's bearer (for example with `createAgentTokenVerifier` from [`@corbits/agent-token`](https://github.com/corbitsdev/corbits-agent-token)), and `resolveRun` maps a run address to the run's tenant and principal from the hub's workflow runs. -```bash -git clone https://github.com/corbitsdev/corbits-memory.git -cd corbits-memory -bun install -bun run typecheck # tsc --noEmit -bun run test # bun test ./src +Add the tools to an agent and bind its `hub` credential to the agent's hub token when you deploy it: + +```ts +import { defineAgent, type InferencePreference } from "@intx/agent"; +import { memory } from "@corbits/memory/sidecar-bundle"; + +export function buildAssistant(sources: readonly InferencePreference[]) { + return defineAgent({ + id: "assistant", + systemPrompt: "You save and recall the team's notes.", + capabilities: [], + inference: { sources }, + tools: [memory], + }); +} ``` -Unit tests use the in-repo `createFakeDocumentStore`/`createFakeSourceProvider` -so the suite runs without Postgres. `bun run build` compiles -`src/` to `dist/`, which is what the package publishes. +## Upgrading from 0.1 + +- `createMemory({ app })` and `registerMemoryRoutes` are replaced by `createMemoryRoutes(deps)`. Mount it with `app.route("/api/tenants/:tenantId/memory", …)`. +- `runMemoryMigrations(databaseUrl, opts)` is now `runMemoryMigrations(dbConfig, { schema, ftsLanguage })`. `ftsLanguage` is required. +- The first run upgrades a 0.1.0 database in place with no data loss and drops the 0.1.0 migration ledger. You cannot roll back to 0.1.0, and 0.1.0 and 0.2.0 replicas must not share a database. +- `@intx/*`, `drizzle-orm`, `hono`, `hono-openapi` and `postgres` are peer dependencies. +- Internal helpers and test fakes are no longer exported from the package root: the share-grant, transform, retention and feed services, the distiller re-exports (import them from `@corbits/memory/distiller`), corroboration scoring, the embed model registry, degrade metrics, the FTS language helpers, `createFakeDocumentStore`/`createFakeSourceProvider`, `createInMemoryWritableGrantStore`/`isWritableGrantStore`, `resolveGrantConfig`/`GrantConfig` and `registerMemoryRoutes`. `runMemoryMigrations` is only exported from `@corbits/memory/migrations`. + +See the [changelog](https://github.com/corbitsdev/corbits-memory/blob/main/CHANGELOG.md) for the full list. ## License -LGPL-2.1-only — see [`LICENSE`](LICENSE). +[LGPL-2.1-only](https://github.com/corbitsdev/corbits-memory/blob/main/LICENSE) diff --git a/src/distiller/workflow.ts b/src/distiller/workflow.ts index 5137027..3cde82c 100644 --- a/src/distiller/workflow.ts +++ b/src/distiller/workflow.ts @@ -8,9 +8,9 @@ * mailTo: "resident-distiller@tenant.example.com", * inference: { sources: [{ provider: "openai", model: "gpt-4.1-mini" }] }, * }); - * // deploy `workflow` and `agent` with host workflow-deploy; the host binds - * // the `hub` credential handle the sidecar bundle resolves at run time, and - * // ticks the workflow by mailing `mailTo`. + * // deploy `workflow` and `agent` with the host's workflow-deploy, binding + * // the `hub` credential the memory tools call the hub with; each mail to + * // `mailTo` runs one distill pass. * ``` */ import {