diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c4eb3e7..38c1b13 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,8 @@ jobs: with: bun-version: latest - run: bun install --frozen-lockfile + - run: bun run lint + - run: bun run format:check - run: bun run typecheck - run: bun run test - run: bun run test:e2e diff --git a/.oxfmtrc.json b/.oxfmtrc.json new file mode 100644 index 0000000..9af2565 --- /dev/null +++ b/.oxfmtrc.json @@ -0,0 +1,4 @@ +{ + "$schema": "./node_modules/oxfmt/configuration_schema.json", + "printWidth": 80 +} diff --git a/.oxlintrc.json b/.oxlintrc.json new file mode 100644 index 0000000..0cd4329 --- /dev/null +++ b/.oxlintrc.json @@ -0,0 +1,6 @@ +{ + "$schema": "./node_modules/oxlint/configuration_schema.json", + "categories": { + "correctness": "error" + } +} diff --git a/AGENTS.md b/AGENTS.md index 0e4be9b..385b2ae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,6 @@ A library, not a service. `src/` is the whole product: a memory **add / search / list** SDK that **mounts onto a host Interchange app**. There is no server, port, or process entrypoint here, and there never should be. - ## Commands ```bash @@ -61,6 +60,7 @@ CI runs `typecheck` + `test` — both must pass before any push. the host's own grant store / `callerResolver` closure, resolved down to `{ tenantId, principalId }` before it ever reaches this package — not in a wider `ResolvedCaller`. + 2. **One Postgres**: `DATABASE_URL`, the engine's own vector plane, under the `memory` schema — never the host's control-plane DB. No foreign keys into control-plane tables; cross-refs (`tenant_id`, `principal_id`) are plain diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index cafe3da..f6c36b3 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -62,16 +62,16 @@ helpers are optional multi-writer / backfill — not the primary path. - **Grants delegate to the host.** Pass `grantStore` + `conditionRegistry`; routes use `createRequireGrant("memory", action)`. - **Two authorization mechanisms, not one — know which is source of truth - for what.** (1) Grant tags decide *capability* (may this principal call - `add`/`search`/`forget`/`purge` at all — `requireGrant`) and *visibility* + for what.** (1) Grant tags decide _capability_ (may this principal call + `add`/`search`/`forget`/`purge` at all — `requireGrant`) and _visibility_ (which documents a principal may see — `accessTags` + `canAccessDocument` in `grant-tags.ts`, where a share grant legitimately widens who can find a document). (2) A separate, imperative **ownership** check — the creator lookup in `services/retention-ownership.ts`, called from `memory.ts` — - decides who may *forget or purge* a specific document, and is the sole + decides who may _forget or purge_ a specific document, and is the sole source of truth for "whose document is this": it is never derived from grant tags and a share grant never satisfies it. `MemoryGrantRequirement. - installHint` (`grant-requirements.ts`) looks adjacent to this but is not: +installHint` (`grant-requirements.ts`) looks adjacent to this but is not: it is advisory metadata for install tooling sizing a capability grant, read by nothing at request time. Do not extend mechanism (1) expecting it to cover ownership — extend `retention-ownership.ts` instead. diff --git a/IMPLEMENTATION.md b/IMPLEMENTATION.md index 26668e4..cd6d3ab 100644 --- a/IMPLEMENTATION.md +++ b/IMPLEMENTATION.md @@ -53,7 +53,6 @@ scripts/db-setup.ts # runs the idempotent migrations against DATABASE_URL compose.yml # pgvector + Ollama + reranker for local dev ``` - The SDK has no server and no process entrypoint. `createMemory` takes the host's `Hono` app plus `{ config, grants? }` and mounts the @@ -85,18 +84,18 @@ There are two config types, both in the SDK: throws if unset/empty, `optionalEnv(name)` returns `undefined`, `intEnv(name, fallback)` parses a positive integer or throws. -| Var | Required? | Default | Notes | -|---|---|---|---| -| `DATABASE_URL` | **yes** | — | the engine's own pgvector Postgres | -| `DB_POOL_MAX` | no | `8` | postgres-js pool size | -| `FTS_LANGUAGE` | no | `english` | text search config for the lexical channel; fixed into the generated column at migration time — changing it later requires rebuilding the column (recipe below), and `runMemoryMigrations` fails loudly if config and column disagree. Unqualified `pg_catalog` config names only — a schema-qualified config (`myschema.mycfg`) is rejected explicitly, both when configuring and when read back from an already-migrated column. | -| `EMBED_BASE_URL` | no | — | embed endpoint root, no path suffix; absent (with `EMBED_MODEL` also absent) => lexical-only, see below | -| `EMBED_MODEL` | no | — | model id/name passed to the embed endpoint; must be set together with `EMBED_BASE_URL` (both or neither — one without the other throws) | -| `EMBED_API_STYLE` | no | `"openai"` | `"openai" \| "tei" \| "ollama"` | -| `EMBED_API_KEY` | no | `undefined` | forwarded as `Authorization: Bearer ` | -| `RERANK_BASE_URL` | no | `undefined` | absent => search degrades to fusion-only | -| `RERANK_MODEL` | no | `undefined` | defaults to `bge-reranker-v2-m3` in the client | -| `RERANK_API_KEY` | no | `undefined` | forwarded as Bearer token to the rerank endpoint | +| Var | Required? | Default | Notes | +| ----------------- | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `DATABASE_URL` | **yes** | — | the engine's own pgvector Postgres | +| `DB_POOL_MAX` | no | `8` | postgres-js pool size | +| `FTS_LANGUAGE` | no | `english` | text search config for the lexical channel; fixed into the generated column at migration time — changing it later requires rebuilding the column (recipe below), and `runMemoryMigrations` fails loudly if config and column disagree. Unqualified `pg_catalog` config names only — a schema-qualified config (`myschema.mycfg`) is rejected explicitly, both when configuring and when read back from an already-migrated column. | +| `EMBED_BASE_URL` | no | — | embed endpoint root, no path suffix; absent (with `EMBED_MODEL` also absent) => lexical-only, see below | +| `EMBED_MODEL` | no | — | model id/name passed to the embed endpoint; must be set together with `EMBED_BASE_URL` (both or neither — one without the other throws) | +| `EMBED_API_STYLE` | no | `"openai"` | `"openai" \| "tei" \| "ollama"` | +| `EMBED_API_KEY` | no | `undefined` | forwarded as `Authorization: Bearer ` | +| `RERANK_BASE_URL` | no | `undefined` | absent => search degrades to fusion-only | +| `RERANK_MODEL` | no | `undefined` | defaults to `bge-reranker-v2-m3` in the client | +| `RERANK_API_KEY` | no | `undefined` | forwarded as Bearer token to the rerank endpoint | **Lexical-only mode (CL-6287).** `EngineConfig.embed` is optional — leave both `EMBED_BASE_URL`/`EMBED_MODEL` unset and the engine still constructs and @@ -155,6 +154,7 @@ each run, so there is no ledger). No memory table has a foreign key into any control-plane table — `tenant_id`/`principal_id`/source refs are plain `text`. ### `memory_document` + The stable logical row for a captured source. Unique on `(tenant_id, adapter, external_ref)` — this triple is the dedupe/identity key every capture upserts against. Document access is **grant tags**: @@ -163,6 +163,7 @@ every capture upserts against. Document access is **grant tags**: `last_seen_at` bumps on every re-capture, even a content-hash NOOP. ### `memory_version` + The versioned body of a document. `version` is a monotonic integer scoped to `(document_id, generation)` — **not** globally per-document — per the `memory_version_document_generation_version_uniq` unique index (baseline @@ -182,6 +183,7 @@ so a replayed corpus's versions never collide with, or even become visible alongside, the live ones unless a caller explicitly searches that generation. ### `memory_chunk` + An ordered slice of a version's text, keyed by `(version_id, ordinal)` (unique). Carries a generated-always `text_fts tsvector` column (GIN-indexed) that powers the lexical search channel — this is the only place FTS is @@ -233,6 +235,7 @@ column is rejected explicitly by `verifyFtsLanguage` (with this same recipe in the error) rather than silently mis-parsed. ### `memory_entity` / `memory_edge` + Lightweight graph rows. `memory_entity` has no unique constraint; dedupe on `(tenant_id, kind, identifiers)` is done in application code (`upsertEntity` in `capture.ts`, an exact-match linear scan per kind). Same for @@ -250,9 +253,9 @@ migration CHECK sets match the TS constants. Two axes on `memory.version`, orthogonal to ranking priors: -| Column / field | Values | Meaning | -| --- | --- | --- | -| `provenance` | `stated` \| `inferred` \| `unknown` | How content was obtained. Capture defaults to `stated`; distilled claims write `inferred`. Existing rows default `unknown`. | +| Column / field | Values | Meaning | +| ------------------------ | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| `provenance` | `stated` \| `inferred` \| `unknown` | How content was obtained. Capture defaults to `stated`; distilled claims write `inferred`. Existing rows default `unknown`. | | `source_class` (lineage) | `native` \| `imported` \| `derived` | Data lineage. Adapters write `native` (or `imported` for bulk import); distilled claims write `derived` via `AdaptedDocument.lineageClass`. | Ranking priors (`AdaptedDocument.sourceClass`: `native|thread|channel|call|record`) feed `computeAuthority` only and are **not** written to the version `source_class` column — that was a latent CHECK violation fixed when the axes were split. @@ -263,16 +266,17 @@ A derived claim is a normal version with `provenance: inferred`, `lineageClass: See `docs/TEMPORAL.md`. On `memory.version`: -| Column | Role | -| --- | --- | -| `occurred_at` | Effective time the content refers to | -| `ingested_at` | When the plane learned it (no separate `asserted_at`) | -| `temporal_class` | `event` \| `deadline` \| `state` \| `lesson` — ranking prior | -| `valid_from` / `valid_until` | Optional validity window | +| Column | Role | +| ---------------------------- | ------------------------------------------------------------ | +| `occurred_at` | Effective time the content refers to | +| `ingested_at` | When the plane learned it (no separate `asserted_at`) | +| `temporal_class` | `event` \| `deadline` \| `state` \| `lesson` — ranking prior | +| `valid_from` / `valid_until` | Optional validity window | Search multiplies fused scores by `temporalRecencyMultiplier` (class-aware). Timeline and search both filter `generation` (default live) so replay rows never leak into live views. ### `memory_embed_model` + Per-tenant registry of embed models and their discovered dimensionality (`discoverModelDims`/`probeEmbedDims` — dims are **never** hard-coded, always probed live against the endpoint). Unique on `(tenant_id, model_key)`, where @@ -297,12 +301,13 @@ Optional `archived_live_generation` and `archived_live_model_key` on `transform_run` record which generation and dense model held live rows before promote (for demote/rollback of both corpus and dense search). - ### Dynamic per-model vector tables: `memory_embedding_` + Not in `db/schema.ts` (no fixed shape — dimensionality varies by model) and not in any migration file. Created at runtime by `activateEmbedModel` (`embed-model-registry.ts`) the first time a given `(baseUrl, modelId)` pair is used: + ```sql CREATE TABLE IF NOT EXISTS memory_embedding_ ( chunk_id text PRIMARY KEY, @@ -312,17 +317,20 @@ CREATE TABLE IF NOT EXISTS memory_embedding_ ( FOREIGN KEY (chunk_id) REFERENCES memory_chunk (id) ON DELETE CASCADE ) ``` + plus a `(tenant_id, chunk_id)` index (created on every activation, so it retrofits onto older tables). The FK completes the hard-delete cascade chain document -> version -> chunk -> embedding. `CREATE TABLE IF NOT EXISTS` cannot add the FK to a table created before it existed; that gap is accepted (no populated pre-FK installs exist). If hard deletes are ever introduced against such a table, run once, per embedding table: + ```sql ALTER TABLE memory_embedding_ ADD CONSTRAINT memory_embedding__chunk_fk FOREIGN KEY (chunk_id) REFERENCES memory_chunk (id) ON DELETE CASCADE; ``` + plus an HNSW index: `vector_cosine_ops` up to 2000 dims (falling back to `ivfflat` if the Postgres/pgvector build lacks the `hnsw` access method), or a `halfvec` expression index (`halfvec_cosine_ops`, pgvector >= 0.7) for @@ -342,6 +350,7 @@ string-interpolated into raw SQL — this is the only place in the codebase a computed identifier is spliced into DDL/DML. ### `raw_capture` + The immutable, append-only substrate (the raw-capture layer). Stores the exact add/ingest request payload (`adapter`, `occurred_at`, `document`) as JSON in `raw_text` (there's also a `raw_bytes bytea` column for non-textual payloads, currently @@ -353,6 +362,7 @@ inserting a duplicate; the table never has rows updated or deleted by ingestion. ### `transform_config` + A named, versioned recipe of derivation + retrieval-tuning knobs (`TransformConfigParams`: `chunk` — only `token.recursive` is a valid strategy today — `embed`, optional `rerank`, `authorityWeight`, @@ -361,6 +371,7 @@ strategy today — `embed`, optional `rerank`, `authorityWeight`, `version = max(existing) + 1` rather than colliding (`createTransformConfig`). ### `transform_run` + One execution of a `transform_config` against a (possibly scoped) slice of `raw_capture`. `generation` is this run's own id, unique (`transform_run_generation_uniq`) — so a generation always resolves back to @@ -381,10 +392,10 @@ returns the run summary either way. defaults regardless of chunker, which is why the live path is unaffected by a replay's custom caps), and computes `contentHash` over `title + kind + externalRef + stableStringify(attributes) + joined chunk - text` (`core/hash.ts`) — this hash is the NOOP-check key. +text` (`core/hash.ts`) — this hash is the NOOP-check key. 2. **`insertOrReuseRawCapture`**: hashes the raw wire payload (`computeSourceHash`, independent of `contentHash` — this one covers the - *unadapted* input including `adapter`/`occurredAt`) and looks it up by + _unadapted_ input including `adapter`/`occurredAt`) and looks it up by `(tenantId, sourceHash)`; reuses the existing `raw_capture` row or inserts a new one, all inside the same transaction as the derived rows. 3. **`deriveVersionInTransaction`** — the single derivation core shared by @@ -490,17 +501,17 @@ and `GET .../memory/feed` pull live versions ordered by `feed_seq` (migration - live generation → `resolveActiveEmbedTable` (tenant active model) - staged generation → `resolveEmbedTableByModelKey` for the transform config's embed model (ready or active; never activates) - runs a raw-SQL cosine-distance ANN query via `cosineDistanceExpr` - (`e.embedding <=> $vector` up to 2000 dims, or the matching - `(e.embedding::halfvec(N)) <=> $vector::halfvec(N)` expression above that - so the halfvec HNSW index is used) - against that table joined back to - `memory_chunk`/`memory_version`/`memory_document` with the - **same tenant-only scope** as the lexical channel (no mini-ACL in SQL). - Returns `null` (not an error) when there's no active embed - model yet or the query is empty; a thrown error from the embed call or - the SQL itself is caught by the caller and also folds into `null`/degraded - — the dense channel never fails the whole search. + runs a raw-SQL cosine-distance ANN query via `cosineDistanceExpr` + (`e.embedding <=> $vector` up to 2000 dims, or the matching + `(e.embedding::halfvec(N)) <=> $vector::halfvec(N)` expression above that + so the halfvec HNSW index is used) + against that table joined back to + `memory_chunk`/`memory_version`/`memory_document` with the + **same tenant-only scope** as the lexical channel (no mini-ACL in SQL). + Returns `null` (not an error) when there's no active embed + model yet or the query is empty; a thrown error from the embed call or + the SQL itself is caught by the caller and also folds into `null`/degraded + — the dense channel never fails the whole search. 4. **RRF fusion** — `fuseRrf` (`core/hybrid-search.ts`): combines the lexical and dense per-channel rank orders (never raw scores — they're on incomparable scales) via Reciprocal Rank Fusion, @@ -508,7 +519,7 @@ and `GET .../memory/feed` pull live versions ordered by `feed_seq` (migration 5. **Per-document dedupe** (non-reranked path only) — `dedupeCandidatesPerDocument`: collapses to the single highest-scoring chunk per `documentId`, using `authorityWeightedScore` (`relevance * (1 + - 0.5 * authority)`) as the rank prior, tie-broken by recency. +0.5 * authority)`) as the rank prior, tie-broken by recency. 6. **Rerank** (only if a rerank endpoint is configured — engine env `RERANK_BASE_URL`, or the replay generation's `transform_config.params.rerank`): dedupe (without the authority prior — authority is applied later on this @@ -529,7 +540,7 @@ and `GET .../memory/feed` pull live versions ordered by `feed_seq` (migration 9. **Degrade path** — if reranking fails (network error, non-2xx) or was never configured, the pipeline falls back to `dedupeCandidatesPerDocument(mergedRows, true, authorityWeight).slice(0, - k)` (fused + authority-weighted order, no MMR) and reports +k)` (fused + authority-weighted order, no MMR) and reports `degraded: ["rerank_unavailable"]`. If dense retrieval failed/unconfigured, `degraded` includes `"dense_unavailable"` and lexical alone answers. 10. **Finishing** — `attachEntityIds` joins in each surviving document's @@ -541,8 +552,8 @@ and `GET .../memory/feed` pull live versions ordered by `feed_seq` (migration the lexical channel contributed zero rows (a dense-only result never reports `"strong"`); otherwise `deriveEvidence` on the lexical rows — `"strong"` requires **both** the top-ranked hit's raw `ts_rank ≥ - STRONG_RANK_FLOOR (0.05)` **and** its authority `≥ - AUTHORITY_STRONG_FLOOR (0.3)`; else `"weak"`. +STRONG_RANK_FLOOR (0.05)` **and** its authority `≥ +AUTHORITY_STRONG_FLOOR (0.3)`; else `"weak"`. Tenant isolation is unconditional and first in every query (`tenant_id` filtered before grant-tag / creator document access); every table/channel is scoped that @@ -553,9 +564,11 @@ way, with no exception. `raw_capture` is the immutable substrate every replay reads from and never writes to. A `transform_config` is a named/versioned recipe (`createTransformConfig`, `listTransformConfigs`) capturing chunk/embed/rerank -+ retrieval-tuning knobs. + +- retrieval-tuning knobs. `runTransform(configId, scope?)`: + 1. Loads the config, mints a new `runId` = the run's own `generation`. 2. Inserts a `transform_run` row (`status: 'running'`). 3. Selects every `raw_capture` row in `scope` (adapter/since/until filters on @@ -602,7 +615,6 @@ active dense embed model for a tenant. The host must treat them as admin operations (grant-gate before calling); never expose them unauthenticated to end-user agents. - ## Mounted routes `createMemoryRoutes(deps)` serves these once the host mounts it at @@ -631,15 +643,15 @@ request payload cap that this package does not implement (see CL-6286's PR body for why) — re-home both as host middleware before deleting the old surface, or a migrating host silently loses them. -| Method + path | Grant action | Request body | Response | -|---|---|---|---| -| `POST /api/tenants/:tenantId/memory/add` | `add` | `{ title, text, access_tags?, share? }` | `200 { documentId, versionId }`; `400` on validation | -| `POST /api/tenants/:tenantId/memory/search` | `search` | `{ query, limit?, kinds?, entity_ids?, sources?, includeEvidence? }` (limit 1–50; `kinds`/`entity_ids`/`sources` narrow retrieval before fusion; unset or `[]` = unfiltered; `includeEvidence` adds a short evidence string when true) | `200 { items[], evidence?, degraded? }`; `400` on bad input | -| `GET /api/tenants/:tenantId/memory/list` | `search` | query `?limit=` (1–100, string on the wire) | `200 { events: [{ at, title, source, tenantId, principalId }] }` — durable recent documents for the caller's scope, filtered with grant-tag access (`canAccessDocument`). One event per document (active live version). | -| `GET /api/tenants/:tenantId/memory/feed` | `search` | query `?after=&limit=&exclude_generator=` | `200 { entries[], nextCursor }` — cursor pull of new live versions. See `docs/FEED.md`. | -| `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` | `forget` | `{ reason? }` | `200 { documentId, versions }`; `403` unless caller is the document's creator; `404` unknown document. Tombstones — content is redacted, not archived; see docs/RETENTION.md. | -| `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` | `purge` | none | `200 { documentId, deleted, reason? }`; `403` unless caller is the document's creator; `404` unknown document. Hard-deletes the row — irreversible; refused while a `durable` version is untombstoned. | -| `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` | `forget` | `{ retention_class }` | `200 { versionId, documentId, status }`; `400` invalid class; `403` unless caller is the version's creator; `404` unknown version. | +| Method + path | Grant action | Request body | Response | +| ------------------------------------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `POST /api/tenants/:tenantId/memory/add` | `add` | `{ title, text, access_tags?, share? }` | `200 { documentId, versionId }`; `400` on validation | +| `POST /api/tenants/:tenantId/memory/search` | `search` | `{ query, limit?, kinds?, entity_ids?, sources?, includeEvidence? }` (limit 1–50; `kinds`/`entity_ids`/`sources` narrow retrieval before fusion; unset or `[]` = unfiltered; `includeEvidence` adds a short evidence string when true) | `200 { items[], evidence?, degraded? }`; `400` on bad input | +| `GET /api/tenants/:tenantId/memory/list` | `search` | query `?limit=` (1–100, string on the wire) | `200 { events: [{ at, title, source, tenantId, principalId }] }` — durable recent documents for the caller's scope, filtered with grant-tag access (`canAccessDocument`). One event per document (active live version). | +| `GET /api/tenants/:tenantId/memory/feed` | `search` | query `?after=&limit=&exclude_generator=` | `200 { entries[], nextCursor }` — cursor pull of new live versions. See `docs/FEED.md`. | +| `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` | `forget` | `{ reason? }` | `200 { documentId, versions }`; `403` unless caller is the document's creator; `404` unknown document. Tombstones — content is redacted, not archived; see docs/RETENTION.md. | +| `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` | `purge` | none | `200 { documentId, deleted, reason? }`; `403` unless caller is the document's creator; `404` unknown document. Hard-deletes the row — irreversible; refused while a `durable` version is untombstoned. | +| `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` | `forget` | `{ retention_class }` | `200 { versionId, documentId, status }`; `400` invalid class; `403` unless caller is the version's creator; `404` unknown version. | `createMemoryRoutes` serves these seven HTTP routes (add, search, list, feed, forget, purge, retention-class). @@ -673,11 +685,11 @@ mountWorkflowMemory(workflowMemoryApp, { app.route("/api/workflow-memory", workflowMemoryApp); ``` -| Constant | Value | -| --- | --- | -| `WORKFLOW_MEMORY_BASE_PATH` | `/api/workflow-memory` | -| `HUB_CREDENTIAL_HANDLE` | `hub` | -| `SIDECAR_BUNDLE_ID` | `@corbits/memory/sidecar-bundle` | +| Constant | Value | +| --------------------------- | -------------------------------- | +| `WORKFLOW_MEMORY_BASE_PATH` | `/api/workflow-memory` | +| `HUB_CREDENTIAL_HANDLE` | `hub` | +| `SIDECAR_BUNDLE_ID` | `@corbits/memory/sidecar-bundle` | **Auth.** Every route sits behind middleware: `agentToken.verify(c)` reads the presented `Authorization`; `agentToken.resolveRun` looks up @@ -690,12 +702,12 @@ token's tenant. No `requireGrant`. No tenant override. Scope on context: are `{ data: … }` on success. Sidecar `requires`: `capabilities`, `address` (run address). Tool results are JSON strings. -| Tool name | Method + path | Notes | -| --- | --- | --- | -| `memory_add` | `POST /add` | Body same as tenant add. Forces `share.tenant = true` (team share; explicit share only widens). Capture path is `memory.add` → `captureDocument`. | -| `memory_search` | `POST /search` | Body same as tenant search; `visibleTags` = `[tenantTag(tenantId)]`. | -| `memory_list` | `GET /list?limit=` | Same grant-tag visibility as search (`visibleTags` team tag). | -| `memory_feed` | `GET /feed?after=&limit=&exclude_generator=` | **501** if `memory.feed` is undefined on this plane. | +| Tool name | Method + path | Notes | +| --------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| `memory_add` | `POST /add` | Body same as tenant add. Forces `share.tenant = true` (team share; explicit share only widens). Capture path is `memory.add` → `captureDocument`. | +| `memory_search` | `POST /search` | Body same as tenant search; `visibleTags` = `[tenantTag(tenantId)]`. | +| `memory_list` | `GET /list?limit=` | Same grant-tag visibility as search (`visibleTags` team tag). | +| `memory_feed` | `GET /feed?after=&limit=&exclude_generator=` | **501** if `memory.feed` is undefined on this plane. | Unknown tool name → tool error (`isError: true`). Non-HTTP hub credential kind → error. `callMemoryRoute` sends `x-workflow-run-address` and @@ -722,20 +734,18 @@ Audience widening uses `splitAudienceWiden` (write-narrow-then-widen) and `shareWidenReceipt` on version attributes after source-owner approval. Ask-on-read remains design-only (fail-closed). - - ### Timeline wire fields (vs the old CaptureLog ring) The process-local CaptureLog hardcoded `source: "api"` and recorded the HTTP caller's `subjectId` as `principalId` at capture time. The durable timeline maps different columns: -| Wire field | Source column / meaning | -|---|---| -| `at` | `memory_document.last_seen_at` (ISO) — re-captures rise in the feed | -| `title` | `memory_document.title` | -| `source` | `memory_document.adapter` (HTTP add defaults to `"http"`, not `"api"`) | -| `tenantId` | `memory_document.tenant_id` | +| Wire field | Source column / meaning | +| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `at` | `memory_document.last_seen_at` (ISO) — re-captures rise in the feed | +| `title` | `memory_document.title` | +| `source` | `memory_document.adapter` (HTTP add defaults to `"http"`, not `"api"`) | +| `tenantId` | `memory_document.tenant_id` | | `principalId` | `memory_version.created_by_principal_id` of the active live version (empty string when null) — the capturing actor stored on the version, not the request principal of a later timeline read | ### Document access (grant tags) @@ -755,7 +765,6 @@ Document access is Interchange authz — **not** a mini-ACL. See `docs/AUTHZ-DOCUMENT-ACCESS.md`. - ## Observability The SDK does no logging or error-reporting setup of its own — that belongs to diff --git a/PRODUCT.md b/PRODUCT.md index 83844e0..ac5ee00 100644 --- a/PRODUCT.md +++ b/PRODUCT.md @@ -15,11 +15,11 @@ package: they carry `@corbits/memory/sidecar-bundle` and hit the parallel add → ingest elements → process (optional) ``` -| Stage | Meaning | Where | -| --- | --- | --- | -| **add** | Something arrives (agent tool, host job, webhook body) | Caller → `memory.add` / `POST …/memory/add` | -| **ingest elements** | Normalize → raw capture → chunks / edges → embed → search-ready | Default `DocumentStore` capture path (sync on `add`) | -| **process** | Optional brain work: classify, claims, links, forget | Host workflow / injected inference — same run as ingest when possible | +| Stage | Meaning | Where | +| ------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------- | +| **add** | Something arrives (agent tool, host job, webhook body) | Caller → `memory.add` / `POST …/memory/add` | +| **ingest elements** | Normalize → raw capture → chunks / edges → embed → search-ready | Default `DocumentStore` capture path (sync on `add`) | +| **process** | Optional brain work: classify, claims, links, forget | Host workflow / injected inference — same run as ingest when possible | Preferred host shape: **one ingest workflow** receives the event, calls `add` (ingest elements), then runs process steps in the same body (or a child step). @@ -34,24 +34,24 @@ or polish when other code also `add`s outside the ingest workflow. See **`src/` is the `@corbits/memory` SDK.** Interchange is the hub — the SDK never creates one; it mounts onto yours. -| Surface | Role | -| --- | --- | -| `createMemory({ … })` | Build the plane | -| `createMemoryRoutes({ memory, requireGrant })` | Hono sub-app the host mounts at `/api/tenants/:tenantId/memory` | -| `mountWorkflowMemory(app, { memory, agentToken })` | Parallel run-scoped `/api/workflow-memory/*` for deployed agents | -| `loadMemoryConfig()` | Config from env | -| `runMemoryMigrations(dbConfig, { schema, ftsLanguage })` | Apply pgvector schema | -| `@corbits/memory/sidecar-bundle` | Deployed-agent factory — no client code, no base URL, no token | -| `@corbits/memory/distiller` | Optional process helpers: `runDistillTick`, `createResidentDistiller` | +| Surface | Role | +| -------------------------------------------------------- | --------------------------------------------------------------------- | +| `createMemory({ … })` | Build the plane | +| `createMemoryRoutes({ memory, requireGrant })` | Hono sub-app the host mounts at `/api/tenants/:tenantId/memory` | +| `mountWorkflowMemory(app, { memory, agentToken })` | Parallel run-scoped `/api/workflow-memory/*` for deployed agents | +| `loadMemoryConfig()` | Config from env | +| `runMemoryMigrations(dbConfig, { schema, ftsLanguage })` | Apply pgvector schema | +| `@corbits/memory/sidecar-bundle` | Deployed-agent factory — no client code, no base URL, no token | +| `@corbits/memory/distiller` | Optional process helpers: `runDistillTick`, `createResidentDistiller` | ### Verbs -| Method | HTTP | Grant | Meaning | -| --- | --- | --- | --- | -| `add` | `POST /api/tenants/:tenantId/memory/add` | `memory:add` | Ingest: capture + derive (chunk/embed on default store) | +| Method | HTTP | Grant | Meaning | +| -------- | ------------------------------------------- | --------------- | ----------------------------------------------------------------------------------- | +| `add` | `POST /api/tenants/:tenantId/memory/add` | `memory:add` | Ingest: capture + derive (chunk/embed on default store) | | `search` | `POST /api/tenants/:tenantId/memory/search` | `memory:search` | Hybrid retrieval (+ optional live sources); hits may include additive `attribution` | -| `list` | `GET /api/tenants/:tenantId/memory/list` | `memory:search` | Recent documents for the principal | -| `feed` | `GET /api/tenants/:tenantId/memory/feed` | `memory:search` | Cursor pull of new live versions (optional multi-writer / backfill) | +| `list` | `GET /api/tenants/:tenantId/memory/list` | `memory:search` | Recent documents for the principal | +| `feed` | `GET /api/tenants/:tenantId/memory/feed` | `memory:search` | Cursor pull of new live versions (optional multi-writer / backfill) | Engine-only plane helpers (no HTTP yet): transform/replay, retention (`deprecateVersion` / `tombstoneDocument` / … — see `docs/RETENTION.md`), @@ -107,10 +107,10 @@ Deployed agent (sidecar-bundle) ### Ports -| Port | Purpose | -| --- | --- | -| `DocumentStore` | Sole durable backend for add/search/list (default: pgvector). Inject fakes or a host/adapter store to skip Postgres. | -| `SourceProvider` | Optional live search merge (fail-soft). Not a store replacement. | +| Port | Purpose | +| ---------------- | -------------------------------------------------------------------------------------------------------------------- | +| `DocumentStore` | Sole durable backend for add/search/list (default: pgvector). Inject fakes or a host/adapter store to skip Postgres. | +| `SourceProvider` | Optional live search merge (fail-soft). Not a store replacement. | Optional sibling packages (not in this tree): Mem0 / Supermemory document stores, Linear tools. Core never imports vendor SDKs. diff --git a/bun.lock b/bun.lock index 63d603e..5f3cc8f 100644 --- a/bun.lock +++ b/bun.lock @@ -20,6 +20,8 @@ "drizzle-orm": "0.45.2", "hono": "4.12.31", "hono-openapi": "1.3.1", + "oxfmt": "0.70.0", + "oxlint": "1.85.0", "postgres": "3.4.9", "typescript": "^5.9.0", }, @@ -121,6 +123,82 @@ "@opentelemetry/semantic-conventions": ["@opentelemetry/semantic-conventions@1.43.0", "", {}, "sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg=="], + "@oxfmt/binding-android-arm-eabi": ["@oxfmt/binding-android-arm-eabi@0.70.0", "", { "os": "android", "cpu": "arm" }, "sha512-Xd7YO4/T2axEj6FTLcj4Why3mTBqFMg+x24xtorT4Lb2+1g82090GH0a/4U1m0pABGYiix2bq1pqkYrmV3f0Sw=="], + + "@oxfmt/binding-android-arm64": ["@oxfmt/binding-android-arm64@0.70.0", "", { "os": "android", "cpu": "arm64" }, "sha512-x9rlMYyKXdgKdYyUJzGsK1ZV8P4di/J32ipzcS6Jet6p9r9UAh28neXIMtdlSaJJycdi61Z4YkcLKLpk8ueFjg=="], + + "@oxfmt/binding-darwin-arm64": ["@oxfmt/binding-darwin-arm64@0.70.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-IUTUPvrBVYy7POh4stXzRdz4IVC/1QSaviCWoyenSlOhGu0X9j5K07vCTM9biLjAA2Zs31l0Rj5vvRpj9n95wA=="], + + "@oxfmt/binding-darwin-x64": ["@oxfmt/binding-darwin-x64@0.70.0", "", { "os": "darwin", "cpu": "x64" }, "sha512-vw745q870oTd6J517O24asoX4/E+eK0nxYIFoedSLqgJ+nI5En7+ZS82iZSHZ69zevQrnOXiyHP01dA+t8xD8w=="], + + "@oxfmt/binding-freebsd-x64": ["@oxfmt/binding-freebsd-x64@0.70.0", "", { "os": "freebsd", "cpu": "x64" }, "sha512-NO14EgSM9dFkcg+MfGPxvsKqXYs9LKaxPrOKXpv1R0rLokGGFDcCq6dBMq18dE4wlpFOovX0UZY2uh1P30O7QA=="], + + "@oxfmt/binding-linux-arm-gnueabihf": ["@oxfmt/binding-linux-arm-gnueabihf@0.70.0", "", { "os": "linux", "cpu": "arm" }, "sha512-139OEhHarj9CYoJ/i9gXlPv4KLBGtLj2toseOWYFf09QwlhklaZk+wW3aOvlqoeZtuayvkoSNVWra7WJ21s3VQ=="], + + "@oxfmt/binding-linux-arm-musleabihf": ["@oxfmt/binding-linux-arm-musleabihf@0.70.0", "", { "os": "linux", "cpu": "arm" }, "sha512-GEh2PY3IWTE0M24eNhTduountANSbWyDmMnzFSQE/nGg/bjPugbUgiGuFu+xdqcQSd/HKSwH80/F2yVVD48yhA=="], + + "@oxfmt/binding-linux-arm64-gnu": ["@oxfmt/binding-linux-arm64-gnu@0.70.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-En5i+UJmZSPxuSf47F2Hl5YOzKB0bicQLnGQkeTCMQ35cWLtbrSwACJKfiLqRZrk05DwSnsJkhBRaM3OURtIaA=="], + + "@oxfmt/binding-linux-arm64-musl": ["@oxfmt/binding-linux-arm64-musl@0.70.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-WWOoV5W9Im3flVwOVrWn/2DUlOF8v5vcCip+kcNuaMpulRCh6nzzt1Su2vcL2F908YJIXNV3HvegbBHuyLwKHg=="], + + "@oxfmt/binding-linux-ppc64-gnu": ["@oxfmt/binding-linux-ppc64-gnu@0.70.0", "", { "os": "linux", "cpu": "ppc64" }, "sha512-YUouneIqW+5n7aE8xx/zeZ6/utr/KH7oykcGoFyd8Uz8uh591T1oKlnoWA3BsRq/ZR42oY1w4MUYvS/0e/MQOA=="], + + "@oxfmt/binding-linux-riscv64-gnu": ["@oxfmt/binding-linux-riscv64-gnu@0.70.0", "", { "os": "linux", "cpu": "none" }, "sha512-iEnMf21S5aGVa4hViDGY8sAQ/AHyCu2JPyrQF8P06wtHhSkD1YJBeT4m/KiGewgf7+a5XCYSCRIPcRQa1xwEoQ=="], + + "@oxfmt/binding-linux-riscv64-musl": ["@oxfmt/binding-linux-riscv64-musl@0.70.0", "", { "os": "linux", "cpu": "none" }, "sha512-91Sdniaj20fQzyMeCxMDzTP4c9s4RB8dGQ308xHhDR0n6U7+1Xq7N9klE7mfXq8iV3lRmIGSXi5X23Hn/0XX/g=="], + + "@oxfmt/binding-linux-s390x-gnu": ["@oxfmt/binding-linux-s390x-gnu@0.70.0", "", { "os": "linux", "cpu": "s390x" }, "sha512-uUV30M6E+2TKKGMaKiwfeL4RZrviHXlUxsrYJ/jFBb+1EZy+pnFT+hF73eeWdzh5NqPOAnw0iiMAIqjqiLZPFg=="], + + "@oxfmt/binding-linux-x64-gnu": ["@oxfmt/binding-linux-x64-gnu@0.70.0", "", { "os": "linux", "cpu": "x64" }, "sha512-ivMcX6kNDPhqtbOaBt/ItFlLlTlXNHLgRuNmxP6Na6UuYXRT10llpJcAPbGeRgjjb3Qzv4jwPp3fB0hui50WNQ=="], + + "@oxfmt/binding-linux-x64-musl": ["@oxfmt/binding-linux-x64-musl@0.70.0", "", { "os": "linux", "cpu": "x64" }, "sha512-w+S+fERxYmlZSyZlJK/U292FjyBoH8cCEj21/tYJX6atX5kNSn+HDkhlFQKT2zcMwUW0uAtUL/bOrlwJZwqRdA=="], + + "@oxfmt/binding-openharmony-arm64": ["@oxfmt/binding-openharmony-arm64@0.70.0", "", { "os": "none", "cpu": "arm64" }, "sha512-Zlom1Xkx257R8bk4ZI4zJsrGno2opknz1+5v5baka3nn4FPyvNSdh8JUL4CdN1S1OWRMvJ9UJQ+RfIqGGCUEfA=="], + + "@oxfmt/binding-win32-arm64-msvc": ["@oxfmt/binding-win32-arm64-msvc@0.70.0", "", { "os": "win32", "cpu": "arm64" }, "sha512-FQgPW5R17vzt7cgrJ8eG/dqX00o2xHsqFeLfw4xzA9FRHpN/DjFo9YDonvIIXGxiEuS9F/jZPnGO+KHNKuCo4Q=="], + + "@oxfmt/binding-win32-ia32-msvc": ["@oxfmt/binding-win32-ia32-msvc@0.70.0", "", { "os": "win32", "cpu": "ia32" }, "sha512-ZfZublNhZ+XBndMiXhkiLlPE+XyGRDa0CweeTL6t1fZypfCh1LTg7e5CvnOeTBunq15MskOcRempumSPGAaCQA=="], + + "@oxfmt/binding-win32-x64-msvc": ["@oxfmt/binding-win32-x64-msvc@0.70.0", "", { "os": "win32", "cpu": "x64" }, "sha512-HlIZEn+WzLQL0DszNzldiRl/DPRCX5R0Vkt6qeUPR1YHwy52hZZo4x6HoTOVmKRP2wUiwPGtKsihNY/f8KRaBg=="], + + "@oxlint/binding-android-arm-eabi": ["@oxlint/binding-android-arm-eabi@1.85.0", "", { "os": "android", "cpu": "arm" }, "sha512-q2KO/Zso9UT+OMn0NF9ywn4E4t0MI3yxiDhNyhsQ7DyQJrC4FhFE4TXOi4bktFnOWXTMds8qZSbpv2XwRaNOBg=="], + + "@oxlint/binding-android-arm64": ["@oxlint/binding-android-arm64@1.85.0", "", { "os": "android", "cpu": "arm64" }, "sha512-SxLN3ALjoT9NNdvpjEevGeHvfzTAFrF0NBYB5tzK7/GtCKMze3j1e/m/X2ozqGj2U9hfGG/dg/OG8vpVK4PiDA=="], + + "@oxlint/binding-darwin-arm64": ["@oxlint/binding-darwin-arm64@1.85.0", "", { "os": "darwin", "cpu": "arm64" }, "sha512-Y/Sup/J4f0f9UGsSd/xyCNTeWL+gepO63GBdEDAfue9nBsnk9zMmnIXx1O6b1V8C90vB5nucYNZ0pbMXAp8zJA=="], + + "@oxlint/binding-darwin-x64": ["@oxlint/binding-darwin-x64@1.85.0", "", { "os": "darwin", "cpu": "x64" }, "sha512-ApOSNC04ynpDTwvBD+//0wyfODRSbEzvRoKpX8teffmc27z8AockwSNeMXGJXn5KP85eahDgR/2llICWLkzcnw=="], + + "@oxlint/binding-freebsd-x64": ["@oxlint/binding-freebsd-x64@1.85.0", "", { "os": "freebsd", "cpu": "x64" }, "sha512-bNrVrCOA/kHky3Tu79IXWXe5bhIgLXfUuUEDHlAGOHUk96MkvDZ1ecaQF19rwstrnaqfP1o9nBTqzIr9+ZHkUg=="], + + "@oxlint/binding-linux-arm-gnueabihf": ["@oxlint/binding-linux-arm-gnueabihf@1.85.0", "", { "os": "linux", "cpu": "arm" }, "sha512-NUrzOJ1s/EqsVvfn2L/1D8Wro2LPIZUbihL8kOJLh5fEdGEN3rdOGUYq3HwnUIL8sjpoP+4N6RaGrgmMJnaMPw=="], + + "@oxlint/binding-linux-arm-musleabihf": ["@oxlint/binding-linux-arm-musleabihf@1.85.0", "", { "os": "linux", "cpu": "arm" }, "sha512-UJXrAT3E/RWkEqXLIs2ehETja1qfgkPb+5gwLIIS+o/6cf+grHvoOXTa5997a/YNQfcJS0DRBTOfZt95cvOI1g=="], + + "@oxlint/binding-linux-arm64-gnu": ["@oxlint/binding-linux-arm64-gnu@1.85.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-lK40QLjI0HxigO7CjDDshEtfYIeiYS0020v5BHFPqN4uuQBQxd2K9LNom2dW15o9F1937quSCRVp4ZsVhdbYdg=="], + + "@oxlint/binding-linux-arm64-musl": ["@oxlint/binding-linux-arm64-musl@1.85.0", "", { "os": "linux", "cpu": "arm64" }, "sha512-c2zbdBwGKreHXwRx3gWBuFGJxLhxgsg6YlZ+3H+RgRusU/UEV9jNwJ3HGYK+nRo0LvBa7mt6Kj86xoVotUo8cw=="], + + "@oxlint/binding-linux-ppc64-gnu": ["@oxlint/binding-linux-ppc64-gnu@1.85.0", "", { "os": "linux", "cpu": "ppc64" }, "sha512-tlt/Hy8lZ97/lCPmCgw/B3k/mwh+BzaIPbPkldZEly7TwLmx0xe2CQcaW2g/rR0dOgS9JNGCZsMEqLhUNMGvaw=="], + + "@oxlint/binding-linux-riscv64-gnu": ["@oxlint/binding-linux-riscv64-gnu@1.85.0", "", { "os": "linux", "cpu": "none" }, "sha512-3tNR9Xey82X0zKuY1d8hJ6Rc9gwRDurmqGLnQZa5xqOXy8/YyiqFXjAtugkKLY82obOlpK1eSiDRlgcNPuxtIg=="], + + "@oxlint/binding-linux-riscv64-musl": ["@oxlint/binding-linux-riscv64-musl@1.85.0", "", { "os": "linux", "cpu": "none" }, "sha512-wbGRd5PqCcjkJFHhZuZ2OBSUQY9czlQsoA/cQQB9JK/L9mC5MQgGoKAh+xd8QjA5V+0D3j+Qd1lAWn1I8zlelA=="], + + "@oxlint/binding-linux-s390x-gnu": ["@oxlint/binding-linux-s390x-gnu@1.85.0", "", { "os": "linux", "cpu": "s390x" }, "sha512-3Sn0kSrE4DPZCWV/8o+n4x3aFZxI9ulMnkYlwCbJ8eUVkwRK2IerohE/A/z3SNbCwoPFOCJmGE5Avrq0rrvdvQ=="], + + "@oxlint/binding-linux-x64-gnu": ["@oxlint/binding-linux-x64-gnu@1.85.0", "", { "os": "linux", "cpu": "x64" }, "sha512-JY2pxxYfB62bAGfejljVCqc44etItehPuAyaeSAdMuEMtwNA00ggMnS66lC1oIhos6oOXUkuU6mZ9bpFh3BqWg=="], + + "@oxlint/binding-linux-x64-musl": ["@oxlint/binding-linux-x64-musl@1.85.0", "", { "os": "linux", "cpu": "x64" }, "sha512-5k74vZ6qJBjBHEOlBk9B/iv68Yu0F1Afw/vvT2ar6OGCqEeXLaSjXz2n/IPCbhLG22UoKoYEJTzpYraRdcp6PA=="], + + "@oxlint/binding-openharmony-arm64": ["@oxlint/binding-openharmony-arm64@1.85.0", "", { "os": "none", "cpu": "arm64" }, "sha512-GbAl5qt5TCkPLXTaIISZJnugrcBhra6rodcXc9jYt620UtdsTt71NlNmJmm0frxzFpd54x/G+MkitEJA8I/BoA=="], + + "@oxlint/binding-win32-arm64-msvc": ["@oxlint/binding-win32-arm64-msvc@1.85.0", "", { "os": "win32", "cpu": "arm64" }, "sha512-kjmws5MK0et2swk4ND85D7NVQyDHw162i6whtZDLUA/lo6FQyBZDcmMRCMcVZcNrAhIaftVb00x9ChGDOjjNJA=="], + + "@oxlint/binding-win32-ia32-msvc": ["@oxlint/binding-win32-ia32-msvc@1.85.0", "", { "os": "win32", "cpu": "ia32" }, "sha512-eSsIJx9n4yxvOqYTZyPEMyEXRmE60XH7xGAU7i0Qbsn1lf6Za3CWJ9aRd82oSFKXaxhp+sA6/yMJVRIpLpna6A=="], + + "@oxlint/binding-win32-x64-msvc": ["@oxlint/binding-win32-x64-msvc@1.85.0", "", { "os": "win32", "cpu": "x64" }, "sha512-pBebIPUpKKhWrhSMWhy8TdAZBewiXnfxmaAGxhzxM1068GagqFaTwgKlU6e+UyJ2sPR+VoHouhXuGJkQjsrDvA=="], + "@standard-community/standard-json": ["@standard-community/standard-json@0.3.5", "", { "peerDependencies": { "@standard-schema/spec": "^1.0.0", "@types/json-schema": "^7.0.15", "@valibot/to-json-schema": "^1.3.0", "arktype": "^2.1.20", "effect": "^3.16.8", "quansync": "^0.2.11", "sury": "^10.0.0", "typebox": "^1.0.17", "valibot": "^1.1.0", "zod": "^3.25.0 || ^4.0.0", "zod-to-json-schema": "^3.24.5" }, "optionalPeers": ["@valibot/to-json-schema", "arktype", "effect", "sury", "typebox", "valibot", "zod", "zod-to-json-schema"] }, "sha512-4+ZPorwDRt47i+O7RjyuaxHRK/37QY/LmgxlGrRrSTLYoFatEOzvqIc85GTlM18SFZ5E91C+v0o/M37wZPpUHA=="], "@standard-community/standard-openapi": ["@standard-community/standard-openapi@0.2.9", "", { "peerDependencies": { "@standard-community/standard-json": "^0.3.5", "@standard-schema/spec": "^1.0.0", "arktype": "^2.1.20", "effect": "^3.17.14", "openapi-types": "^12.1.3", "sury": "^10.0.0", "typebox": "^1.0.0", "valibot": "^1.1.0", "zod": "^3.25.0 || ^4.0.0", "zod-openapi": "^4" }, "optionalPeers": ["arktype", "effect", "sury", "typebox", "valibot", "zod", "zod-openapi"] }, "sha512-htj+yldvN1XncyZi4rehbf9kLbu8os2Ke/rfqoZHCMHuw34kiF3LP/yQPdA0tQ940y8nDq3Iou8R3wG+AGGyvg=="], @@ -311,6 +389,10 @@ "openapi-types": ["openapi-types@12.1.3", "", {}, "sha512-N4YtSYJqghVu4iek2ZUvcN/0aqH1kRDuNqzcycDxhOUpg7GdvLa2F3DgS6yBNhInhv2r/6I0Flkn7CqL8+nIcw=="], + "oxfmt": ["oxfmt@0.70.0", "", { "dependencies": { "tinypool": "2.1.2" }, "optionalDependencies": { "@oxfmt/binding-android-arm-eabi": "0.70.0", "@oxfmt/binding-android-arm64": "0.70.0", "@oxfmt/binding-darwin-arm64": "0.70.0", "@oxfmt/binding-darwin-x64": "0.70.0", "@oxfmt/binding-freebsd-x64": "0.70.0", "@oxfmt/binding-linux-arm-gnueabihf": "0.70.0", "@oxfmt/binding-linux-arm-musleabihf": "0.70.0", "@oxfmt/binding-linux-arm64-gnu": "0.70.0", "@oxfmt/binding-linux-arm64-musl": "0.70.0", "@oxfmt/binding-linux-ppc64-gnu": "0.70.0", "@oxfmt/binding-linux-riscv64-gnu": "0.70.0", "@oxfmt/binding-linux-riscv64-musl": "0.70.0", "@oxfmt/binding-linux-s390x-gnu": "0.70.0", "@oxfmt/binding-linux-x64-gnu": "0.70.0", "@oxfmt/binding-linux-x64-musl": "0.70.0", "@oxfmt/binding-openharmony-arm64": "0.70.0", "@oxfmt/binding-win32-arm64-msvc": "0.70.0", "@oxfmt/binding-win32-ia32-msvc": "0.70.0", "@oxfmt/binding-win32-x64-msvc": "0.70.0" }, "peerDependencies": { "svelte": "^5.0.0", "vite-plus": "*" }, "optionalPeers": ["svelte", "vite-plus"], "bin": { "oxfmt": "bin/oxfmt" } }, "sha512-IsHxZ4y0wQLLMhnrJblBJgZsLDzfULrJnAw5j/QqsTlMa/m3AqsbToi+W71uhBGaqlqq/PbbjvHc09TJwdv3Tw=="], + + "oxlint": ["oxlint@1.85.0", "", { "optionalDependencies": { "@oxlint/binding-android-arm-eabi": "1.85.0", "@oxlint/binding-android-arm64": "1.85.0", "@oxlint/binding-darwin-arm64": "1.85.0", "@oxlint/binding-darwin-x64": "1.85.0", "@oxlint/binding-freebsd-x64": "1.85.0", "@oxlint/binding-linux-arm-gnueabihf": "1.85.0", "@oxlint/binding-linux-arm-musleabihf": "1.85.0", "@oxlint/binding-linux-arm64-gnu": "1.85.0", "@oxlint/binding-linux-arm64-musl": "1.85.0", "@oxlint/binding-linux-ppc64-gnu": "1.85.0", "@oxlint/binding-linux-riscv64-gnu": "1.85.0", "@oxlint/binding-linux-riscv64-musl": "1.85.0", "@oxlint/binding-linux-s390x-gnu": "1.85.0", "@oxlint/binding-linux-x64-gnu": "1.85.0", "@oxlint/binding-linux-x64-musl": "1.85.0", "@oxlint/binding-openharmony-arm64": "1.85.0", "@oxlint/binding-win32-arm64-msvc": "1.85.0", "@oxlint/binding-win32-ia32-msvc": "1.85.0", "@oxlint/binding-win32-x64-msvc": "1.85.0" }, "peerDependencies": { "oxlint-tsgolint": ">=7.0.2001", "vite-plus": "*" }, "optionalPeers": ["oxlint-tsgolint", "vite-plus"], "bin": { "oxlint": "bin/oxlint" } }, "sha512-bc26s97nuvPj1ViyPsqmKecVkUWFMEdtayO8MaQ6oiLfs1pj94cQlZZhrh4BPNlr9HQosjhIlwgZKsfcwmcNgg=="], + "p-map": ["p-map@7.0.6", "", {}, "sha512-I4Prw6ivkd6p8PiYR1tXASOAOBzIJwu0TB7fqaX0c/8c3QAehNYmX57EijyGGGBt3c/BIowGwV03RVBtXvHEVg=="], "pako": ["pako@1.0.11", "", {}, "sha512-4hLB8Py4zZce5s4yd9XzopqwVv/yGNhV1Bl8NTmCq1763HeK2+EwVTv+leGeL13Dnh2wfbqowVPXCIO0z4taYw=="], @@ -361,6 +443,8 @@ "tar": ["tar@7.5.21", "", { "dependencies": { "@isaacs/fs-minipass": "^4.0.0", "chownr": "^3.0.0", "minipass": "^7.1.2", "minizlib": "^3.1.0", "yallist": "^5.0.0" } }, "sha512-XdhtCvlMywwxpCW8YEq3lOXBJpUPTR2OHHcwLPO3HwsJqOHa2Ok/oJ7ruGzp+JrKoRPVCzJwAdEjqLW/vNRPHA=="], + "tinypool": ["tinypool@2.1.2", "", {}, "sha512-9YodfrxS9g9IbFr/KOjE5bAeJ0p61n3bW6mqvy0jtoeKd1kTW1Cxm0oulm6KX2lyM9Gl6WIe8nEbY7LWv5ZJww=="], + "to-buffer": ["to-buffer@1.2.2", "", { "dependencies": { "isarray": "^2.0.5", "safe-buffer": "^5.2.1", "typed-array-buffer": "^1.0.3" } }, "sha512-db0E3UJjcFhpDhAF4tLo03oli3pwl3dbnzXOUIlRKrp+ldk/VUxzpWYZENsw2SZiuBjHAk7DfB0VU7NKdpb6sw=="], "typed-array-buffer": ["typed-array-buffer@1.0.3", "", { "dependencies": { "call-bound": "^1.0.3", "es-errors": "^1.3.0", "is-typed-array": "^1.1.14" } }, "sha512-nAYYwfY3qnzX30IkA6AQZjVbtK6duGontcQm1WSG1MD94YLqK0515GNApXkoxKOWMusVssAHWLh9SeaoefYFGw=="], diff --git a/docs/AUTHZ-DOCUMENT-ACCESS.md b/docs/AUTHZ-DOCUMENT-ACCESS.md index d66c6bb..8317b8f 100644 --- a/docs/AUTHZ-DOCUMENT-ACCESS.md +++ b/docs/AUTHZ-DOCUMENT-ACCESS.md @@ -3,14 +3,12 @@ **Status:** grant tags + creator (baseline migrations) **Problem (historical):** an earlier mini-ACL (`visibility` mode + principal list + block list) ran parallel to Interchange grants. That path is gone — document access is grant tags only. - - ## Source of truth -| Layer | Owner | Mechanism | -| --- | --- | --- | -| Who is the caller? | Host (Interchange) | `principal` + `tenant` on context / plane args | -| May they use memory at all? | Host grant store | `authorize(…, resource: "memory", action: "add" \| "search")` | +| Layer | Owner | Mechanism | +| --------------------------------- | --------------------------------------- | ------------------------------------------------------------------------ | +| Who is the caller? | Host (Interchange) | `principal` + `tenant` on context / plane args | +| May they use memory at all? | Host grant store | `authorize(…, resource: "memory", action: "add" \| "search")` | | Which **documents** may they see? | Host grant store + tags on the document | `authorize(…, resource: , action: "search")` for any tag on the doc | There is **one** authorization system: `@intx/authz` + host `GrantStore`. Corbits Memory does not invent modes, allowlists, or block lists as a security boundary. @@ -34,14 +32,16 @@ When the caller does not pass tags/share: grant of `search` on that owner resource (or a matching pattern) — the engine never auto-grants tags to anyone. - ### Explicit tags ```ts plane.add({ - tenantId, principalId, title, text, + tenantId, + principalId, + title, + text, accessTags: ["memory.space:eng", "memory.project:ke"], -}) +}); ``` Host issues grants such as: @@ -64,12 +64,12 @@ Patterns work: a grant on `memory.space:*` matches `memory.space:eng` via `@intx Share helpers **must not** reintroduce visibility modes. They only mint tags: -| Sugar | Tags written | -| --- | --- | -| (omit `share` — owner-only default) | `memory.owner:` | -| `share: { tenant: true }` | `memory.owner:`, `memory.tenant:` | -| `share: { principals: ["p2","p3"] }` | `memory.owner:`, `memory.owner:p2`, `memory.owner:p3` | -| `share: { tags: ["memory.space:eng"] }` | `memory.owner:`, plus those tags | +| Sugar | Tags written | +| --------------------------------------- | ------------------------------------------------------------- | +| (omit `share` — owner-only default) | `memory.owner:` | +| `share: { tenant: true }` | `memory.owner:`, `memory.tenant:` | +| `share: { principals: ["p2","p3"] }` | `memory.owner:`, `memory.owner:p2`, `memory.owner:p3` | +| `share: { tags: ["memory.space:eng"] }` | `memory.owner:`, plus those tags | There is **no** `share.private` key. Owner-only is the default when `share` is omitted. @@ -121,7 +121,7 @@ Deny is expressed as **absence of allow** (or an explicit deny grant in the host ### Capability (unchanged) ```ts -authorize(grantStore, principalId, tenantId, "memory", "search"|"add") +authorize(grantStore, principalId, tenantId, "memory", "search" | "add"); // effect must be "allow" ``` @@ -161,7 +161,11 @@ Unchanged intentional tradeoff: live `SourceProvider` hits are **enrichment unde "title": "…", "text": "…", "access_tags": ["memory.space:eng"], - "share": { "tenant": true, "principals": ["alice"], "tags": ["memory.space:eng"] } + "share": { + "tenant": true, + "principals": ["alice"], + "tags": ["memory.space:eng"] + } } ``` @@ -169,13 +173,12 @@ Identity never in body. `access_tags` and `share` are optional; default owner-on `search` / `list` need no ACL body — principal from context + grant store. - ## Schema Document access is stored as: -| Column | Role | -| --- | --- | +| Column | Role | +| -------------------- | ------------------------------------------------------------------ | | `access_tags text[]` | Resource strings in grant-pattern space (+ creator always allowed) | There is no `visibility_mode`, principal-id array, block list, or dual-write ACL diff --git a/docs/DISTILLER.md b/docs/DISTILLER.md index 60f538c..c65b887 100644 --- a/docs/DISTILLER.md +++ b/docs/DISTILLER.md @@ -27,10 +27,10 @@ inference; tools only need `memory_add` / `memory_search` (and grants). Helpers still useful in-process: -| Export | Use | -| --- | --- | -| `buildDistilledClaim` | Wire body with `generator_agent_id`, `provenance=inferred`, `derived_from` | -| `RESIDENT_DISTILLER_AGENT_ID` | Stable generator id if you write claims | +| Export | Use | +| ----------------------------- | -------------------------------------------------------------------------- | +| `buildDistilledClaim` | Wire body with `generator_agent_id`, `provenance=inferred`, `derived_from` | +| `RESIDENT_DISTILLER_AGENT_ID` | Stable generator id if you write claims | ## Optional: multi-writer / backfill (`runDistillTick`) @@ -89,14 +89,14 @@ on a schedule, e.g. from `@corbits/cron`. This package owns no clock. ## Substrate (plane) -| Piece | Where | -| --- | --- | -| Ingest on add | capture path — raw + chunks + embed | -| Capture feed (cursor) | `memory.feed` — [FEED.md](./FEED.md) (backfill / multi-writer) | -| Claim identity on add | `generator_agent_id`, `provenance`, `lineage_class`, `derived_from` | -| Wire attribution on search | `SearchItem.attribution` | -| Retention / forgetting | [RETENTION.md](./RETENTION.md) | -| Tools | `@corbits/memory/sidecar-bundle` | +| Piece | Where | +| -------------------------- | ------------------------------------------------------------------- | +| Ingest on add | capture path — raw + chunks + embed | +| Capture feed (cursor) | `memory.feed` — [FEED.md](./FEED.md) (backfill / multi-writer) | +| Claim identity on add | `generator_agent_id`, `provenance`, `lineage_class`, `derived_from` | +| Wire attribution on search | `SearchItem.attribution` | +| Retention / forgetting | [RETENTION.md](./RETENTION.md) | +| Tools | `@corbits/memory/sidecar-bundle` | ## Grant manifest (process principal) diff --git a/docs/FEED.md b/docs/FEED.md index e882c41..5ec7256 100644 --- a/docs/FEED.md +++ b/docs/FEED.md @@ -19,10 +19,10 @@ Use the feed when: memory.feed({ tenantId, principalId, after?, limit?, excludeGenerator? }) ``` -| Field | Meaning | -|-------|---------| -| `after` | Last consumed `feedSeq` (exclusive). Omit/0 = from start. | -| `limit` | Page size (bounded). | +| Field | Meaning | +| ------------------ | --------------------------------------------------------- | +| `after` | Last consumed `feedSeq` (exclusive). Omit/0 = from start. | +| `limit` | Page size (bounded). | | `excludeGenerator` | Skip versions with this `generator_agent_id` (loop-safe). | - Ordered by `feed_seq` ascending (Postgres `bigserial` on `memory.version`). @@ -30,7 +30,7 @@ memory.feed({ tenantId, principalId, after?, limit?, excludeGenerator? }) - Capability: `memory` / `search` (same as list/retrieve). - Document access: grant-tag post-filter identical to search. - **Cursor advances past the examined raw page**, even when the access filter - returns zero entries. Using the last *allowed* `feedSeq` would stall a + returns zero entries. Using the last _allowed_ `feedSeq` would stall a consumer on a fully denied page forever. Cursor storage is the **consumer's** job (workflow run state). diff --git a/docs/RELEVANCY.md b/docs/RELEVANCY.md index f230778..6ab6297 100644 --- a/docs/RELEVANCY.md +++ b/docs/RELEVANCY.md @@ -6,9 +6,9 @@ a frozen snapshot (`computeAuthority` at write). Search derives a separate ## Edges -| `rel` | Effect | -|-------|--------| -| `supports` | Independent source backs the target claim version — factor up | +| `rel` | Effect | +| ------------- | --------------------------------------------------------------- | +| `supports` | Independent source backs the target claim version — factor up | | `contradicts` | Disagreement signal — factor down; **no** auto-delete/supersede | Counts are edges with `to_type = 'version'` and `to_ref = `. diff --git a/docs/RETENTION.md b/docs/RETENTION.md index b722f19..2fa5623 100644 --- a/docs/RETENTION.md +++ b/docs/RETENTION.md @@ -3,12 +3,12 @@ Versions carry a **retention class** orthogonal to temporal ranking class (`temporal_class`) and lineage (`source_class` / `provenance`). -| Class | Intent | -| --- | --- | -| `durable` | Long-lived claims; hard-delete blocked until tombstoned | -| `standard` | Default working memory | -| `ephemeral` | Short TTL; sweeper deprecates past `valid_until` (or 7d from `ingested_at`) — hard delete is a separate explicit step | -| `source_only` | Keep raw capture; derived versions may be dropped by host policy | +| Class | Intent | +| ------------- | --------------------------------------------------------------------------------------------------------------------- | +| `durable` | Long-lived claims; hard-delete blocked until tombstoned | +| `standard` | Default working memory | +| `ephemeral` | Short TTL; sweeper deprecates past `valid_until` (or 7d from `ingested_at`) — hard delete is a separate explicit step | +| `source_only` | Keep raw capture; derived versions may be dropped by host policy | Schema: `memory.version.retention_class` (migration `0007_retention.sql`). CHECK constraint `version_retention_class_check` stays lockstep with @@ -16,13 +16,13 @@ CHECK constraint `version_retention_class_check` stays lockstep with ## Write paths -| Verb | Plane API | Effect | -| --- | --- | --- | -| Deprecate | `memory.deprecateVersion` | `status=deprecated`, `deprecated_at` / reason | -| Tombstone | `memory.tombstoneDocument` | All active/deprecated/superseded versions → `tombstoned`; chunk text redacted to `[redacted]` | -| Hard delete | `memory.hardDeleteDocument` | Deletes document row (cascade); **refuses** if any non-tombstoned version is `durable` | -| Sweep | `memory.sweepEphemeral` | Auto-**deprecates** (never deletes) ephemeral versions past `valid_until` (or 7d from `ingested_at`); host schedules, core is cron-free | -| Set class | `memory.setRetentionClass` | Update `retention_class` on a version | +| Verb | Plane API | Effect | +| ----------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| Deprecate | `memory.deprecateVersion` | `status=deprecated`, `deprecated_at` / reason | +| Tombstone | `memory.tombstoneDocument` | All active/deprecated/superseded versions → `tombstoned`; chunk text redacted to `[redacted]` | +| Hard delete | `memory.hardDeleteDocument` | Deletes document row (cascade); **refuses** if any non-tombstoned version is `durable` | +| Sweep | `memory.sweepEphemeral` | Auto-**deprecates** (never deletes) ephemeral versions past `valid_until` (or 7d from `ingested_at`); host schedules, core is cron-free | +| Set class | `memory.setRetentionClass` | Update `retention_class` on a version | Search and feed exclude non-active (and non-superseded for feed) rows by default. Pass `includeDeprecated: true` on search to retrieve deprecated @@ -33,11 +33,11 @@ Service module: `src/services/retention.ts`. ## HTTP surface (CL-6288) -| Route | Grant action | Plane verb | -| --- | --- | --- | -| `POST …/memory/documents/:documentId/forget` | `memory:forget` | `tombstoneDocument` | -| `POST …/memory/documents/:documentId/purge` | `memory:purge` | `hardDeleteDocument` | -| `POST …/memory/versions/:versionId/retention-class` | `memory:forget` | `setRetentionClass` | +| Route | Grant action | Plane verb | +| --------------------------------------------------- | --------------- | -------------------- | +| `POST …/memory/documents/:documentId/forget` | `memory:forget` | `tombstoneDocument` | +| `POST …/memory/documents/:documentId/purge` | `memory:purge` | `hardDeleteDocument` | +| `POST …/memory/versions/:versionId/retention-class` | `memory:forget` | `setRetentionClass` | `deprecateVersion` and `sweepEphemeral` have no route (see below). @@ -50,14 +50,14 @@ un-tombstone/restore verb, and only version metadata (status, timestamps, retention class) remains for audit. `purge` (hard delete) goes further and removes the document row itself; it has its own grant action and is refused outright while a `durable`-class version on the document is untombstoned. The -distinction that matters is *what's still queryable*: after `forget` a +distinction that matters is _what's still queryable_: after `forget` a document row and its metadata still exist (for audit) but its content is gone; after `purge` nothing does. A host can grant `forget` broadly (every user gets a "forget this" button) while keeping `purge` to an operator role — but should not describe `forget` to end users as reversible. **Ownership, not just visibility.** `memory:search`/a document's `accessTags` -say who can *see* a document — never who may forget or purge it. Every +say who can _see_ a document — never who may forget or purge it. Every retention route additionally checks that the caller is the document's creator (`created_by_principal_id` — the document's first version for `forget`/`purge`, the specific version's own creator for `retention-class`), diff --git a/docs/TEMPORAL.md b/docs/TEMPORAL.md index d024208..46e7738 100644 --- a/docs/TEMPORAL.md +++ b/docs/TEMPORAL.md @@ -5,24 +5,24 @@ in capture (write), hybrid-search (rank), and search/timeline (query filters). ## Four times (two stored, two derived) -| Concept | Column / source | Meaning | -| --- | --- | --- | -| **Effective time** | `occurred_at` (required) | When the content *refers to*: event moment, state effective time, or when a deadline was established. Not dual-meaning — one meaning applied across classes. | -| **Ingestion / assertion** | `ingested_at` | When the memory plane learned the content (capture or distill write). There is no separate `asserted_at`. | -| **Validity start** | `valid_from` (nullable) | Optional window start for state/deadline claims. | -| **Validity end** | `valid_until` (nullable) | Optional window end. Required in practice for useful `deadline` ranking. | +| Concept | Column / source | Meaning | +| ------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Effective time** | `occurred_at` (required) | When the content _refers to_: event moment, state effective time, or when a deadline was established. Not dual-meaning — one meaning applied across classes. | +| **Ingestion / assertion** | `ingested_at` | When the memory plane learned the content (capture or distill write). There is no separate `asserted_at`. | +| **Validity start** | `valid_from` (nullable) | Optional window start for state/deadline claims. | +| **Validity end** | `valid_until` (nullable) | Optional window end. Required in practice for useful `deadline` ranking. | ## `temporal_class` SSOT: `TEMPORAL_CLASSES` in `src/core/enums.ts`. Stored on **version** (not document) so successive versions can change class. -| Class | Recency prior | Notes | -| --- | --- | --- | -| `event` | Exponential decay from `occurred_at` (30-day half-life default) | Default for raw captures; preserves pre-model ranking. | -| `deadline` | Neutral far out; urgency ramp in a 7-day lookahead before `valid_until`; floor (0.7) after expiry | Still history-retrievable after expiry — not deleted. | -| `state` | Constant 1.0 while `status='active'` | Default for `provenance='inferred'` (distilled claims). Supersede via capture status, not recency. | -| `lesson` | Constant 1.0 | Always explicit; never a default. | +| Class | Recency prior | Notes | +| ---------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| `event` | Exponential decay from `occurred_at` (30-day half-life default) | Default for raw captures; preserves pre-model ranking. | +| `deadline` | Neutral far out; urgency ramp in a 7-day lookahead before `valid_until`; floor (0.7) after expiry | Still history-retrievable after expiry — not deleted. | +| `state` | Constant 1.0 while `status='active'` | Default for `provenance='inferred'` (distilled claims). Supersede via capture status, not recency. | +| `lesson` | Constant 1.0 | Always explicit; never a default. | Defaults at write: diff --git a/e2e/caller-resolver.test.ts b/e2e/caller-resolver.test.ts index dec784e..a4f8387 100644 --- a/e2e/caller-resolver.test.ts +++ b/e2e/caller-resolver.test.ts @@ -19,187 +19,227 @@ import { // A machine caller (e.g. a workflow run) never passes the host's session // middleware; the host's callerResolver names its tenant and principal. -describe.skipIf(testDatabaseUrl() === undefined)("machine callers through callerResolver", () => { - let db: TestDb; - let memory: Memory | undefined; - let grantStore: GrantStore; - - beforeAll(async () => { - db = await createTestDb(); - await seedPrincipal(db, "acme", "run"); - await seedPrincipal(db, "acme", "other-run"); - await seedPrincipal(db, "globex", "globex-user"); - grantStore = createInMemoryGrantStore([ - ...["add", "search", "forget", "purge"].map((a) => allow("run", a)), - ...["add", "search", "forget"].map((a) => allow("other-run", a)), - ...["add", "search"].map((a) => allow("globex-user", a)), - // Would expose globex's tenant-shared documents if a route ever - // scoped a read to the URL's tenant instead of the resolved one. - { - id: "g-run-globex-tag", - resource: "memory.tenant:globex", - action: "search", - effect: "allow", - origin: "role", - conditions: null, - expiresAt: null, - roleId: null, - principalId: "run", - }, - allow("", "add"), - allow("\t\n", "add"), - ]); - memory = createTestMemory(db, grantStore); - }); - - afterAll(async () => { - await memory?.close(); - await db?.close(); - }); - - function appFor(callerResolver: CallerResolver): Hono { - const app = new Hono(); - app.route( - "/api/tenants/:tenantId/memory", - createMemoryRoutes({ - memory: memory as Memory, - requireGrant: createRequireGrant({ grantStore, conditionRegistry: {} }), - callerResolver, - }), - ); - return app; - } - - const as = (tenantId: string, principalId: string) => - appFor(() => ({ tenantId, principalId })); - - function post(app: Hono, path: string, body: unknown = {}) { - return app.request(`/api/tenants/ignored/memory${path}`, { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify(body), +describe.skipIf(testDatabaseUrl() === undefined)( + "machine callers through callerResolver", + () => { + let db: TestDb; + let memory: Memory | undefined; + let grantStore: GrantStore; + + beforeAll(async () => { + db = await createTestDb(); + await seedPrincipal(db, "acme", "run"); + await seedPrincipal(db, "acme", "other-run"); + await seedPrincipal(db, "globex", "globex-user"); + grantStore = createInMemoryGrantStore([ + ...["add", "search", "forget", "purge"].map((a) => allow("run", a)), + ...["add", "search", "forget"].map((a) => allow("other-run", a)), + ...["add", "search"].map((a) => allow("globex-user", a)), + // Would expose globex's tenant-shared documents if a route ever + // scoped a read to the URL's tenant instead of the resolved one. + { + id: "g-run-globex-tag", + resource: "memory.tenant:globex", + action: "search", + effect: "allow", + origin: "role", + conditions: null, + expiresAt: null, + roleId: null, + principalId: "run", + }, + allow("", "add"), + allow("\t\n", "add"), + ]); + memory = createTestMemory(db, grantStore); }); - } - - async function add(app: Hono, title: string): Promise { - const res = await post(app, "/add", { title, text: `${title} body` }); - expect(res.status).toBe(200); - return ((await res.json()) as { documentId: string }).documentId; - } - - test("add writes under the resolved scope, ignoring identity in the body", async () => { - const res = await post(as("acme", "run"), "/add", { - title: "Run note", - text: "written by the run", - tenantId: "globex", - principalId: "globex-user", + + afterAll(async () => { + await memory?.close(); + await db?.close(); }); - expect(res.status).toBe(200); - const { documentId } = (await res.json()) as { documentId: string }; - const [row] = await db.sql<{ tenant_id: string; created_by_principal_id: string }[]>` + + function appFor(callerResolver: CallerResolver): Hono { + const app = new Hono(); + app.route( + "/api/tenants/:tenantId/memory", + createMemoryRoutes({ + memory: memory as Memory, + requireGrant: createRequireGrant({ + grantStore, + conditionRegistry: {}, + }), + callerResolver, + }), + ); + return app; + } + + const as = (tenantId: string, principalId: string) => + appFor(() => ({ tenantId, principalId })); + + function post(app: Hono, path: string, body: unknown = {}) { + return app.request(`/api/tenants/ignored/memory${path}`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + }); + } + + async function add(app: Hono, title: string): Promise { + const res = await post(app, "/add", { title, text: `${title} body` }); + expect(res.status).toBe(200); + return ((await res.json()) as { documentId: string }).documentId; + } + + test("add writes under the resolved scope, ignoring identity in the body", async () => { + const res = await post(as("acme", "run"), "/add", { + title: "Run note", + text: "written by the run", + tenantId: "globex", + principalId: "globex-user", + }); + expect(res.status).toBe(200); + const { documentId } = (await res.json()) as { documentId: string }; + const [row] = await db.sql< + { tenant_id: string; created_by_principal_id: string }[] + >` SELECT d.tenant_id, v.created_by_principal_id FROM memory.document d JOIN memory.version v ON v.document_id = d.id WHERE d.id = ${documentId}`; - expect(row).toEqual({ tenant_id: "acme", created_by_principal_id: "run" }); - }); + expect(row).toEqual({ + tenant_id: "acme", + created_by_principal_id: "run", + }); + }); - async function documentCount(): Promise { - const [row] = await db.sql<{ n: number }[]>` + async function documentCount(): Promise { + const [row] = await db.sql<{ n: number }[]>` SELECT count(*)::int AS n FROM memory.document`; - return row?.n ?? 0; - } - - test("the resolved caller still needs the grant on every route", async () => { - await seedPrincipal(db, "acme", "ungranted"); - const ungranted = as("acme", "ungranted"); - expect((await post(ungranted, "/add", { title: "t", text: "b" })).status).toBe(403); - expect((await post(ungranted, "/search", { query: "q" })).status).toBe(403); - expect((await ungranted.request("/api/tenants/acme/memory/list")).status).toBe(403); - expect((await ungranted.request("/api/tenants/acme/memory/feed")).status).toBe(403); - }); - - test("a resolver that cannot identify the caller is 401", async () => { - const res = await post(appFor(() => null), "/add", { title: "t", text: "b" }); - expect(res.status).toBe(401); - expect(((await res.json()) as { error: { code: string } }).error.code).toBe( - "unauthorized", - ); - }); - - test("a resolver that throws fails closed without leaking its message", async () => { - const before = await documentCount(); - const res = await post( - appFor(() => { - throw new Error("db connection string with secret=abc123"); - }), - "/add", - { title: "t", text: "b" }, - ); - expect(res.status).toBe(500); - expect(await res.text()).not.toContain("secret=abc123"); - expect(await documentCount()).toBe(before); - }); - - test.each([ - ["empty", { tenantId: "", principalId: "" }], - ["whitespace", { tenantId: " ", principalId: "\t\n" }], - ] as const)("a resolver returning a %s identity is a 500, never a seated scope", async (_label, resolved) => { - // The malformed principal holds an add grant, so a 500 proves the - // identity was rejected before the grant check. - const before = await documentCount(); - const res = await post(appFor(() => resolved), "/add", { title: "t", text: "b" }); - expect(await documentCount()).toBe(before); - expect(res.status).toBe(500); - expect(((await res.json()) as { error: { code: string } }).error.code).toBe( - "invalid_resolved_caller", - ); - }); + return row?.n ?? 0; + } - test("search, list and feed read only the resolved tenant, whatever the URL says", async () => { - const globexShare = await post(as("globex", "globex-user"), "/add", { - title: "Globex secret plan", - text: "globex secret plan body", - share: { tenant: true }, - }); - expect(globexShare.status).toBe(200); - await add(as("acme", "run"), "Acme run plan"); - const acme = as("acme", "run"); - - const search = await acme.request("/api/tenants/globex/memory/search", { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ query: "plan" }), + test("the resolved caller still needs the grant on every route", async () => { + await seedPrincipal(db, "acme", "ungranted"); + const ungranted = as("acme", "ungranted"); + expect( + (await post(ungranted, "/add", { title: "t", text: "b" })).status, + ).toBe(403); + expect((await post(ungranted, "/search", { query: "q" })).status).toBe( + 403, + ); + expect( + (await ungranted.request("/api/tenants/acme/memory/list")).status, + ).toBe(403); + expect( + (await ungranted.request("/api/tenants/acme/memory/feed")).status, + ).toBe(403); }); - expect(search.status).toBe(200); - const { items } = (await search.json()) as { items: { title: string }[] }; - expect(items.map((i) => i.title)).toContain("Acme run plan"); - expect(items.map((i) => i.title)).not.toContain("Globex secret plan"); - for (const route of ["list", "feed"]) { - const res = await acme.request(`/api/tenants/globex/memory/${route}`); - expect(res.status).toBe(200); - const body = JSON.stringify(await res.json()); - expect(body).toContain("Acme run plan"); - expect(body).not.toContain("Globex secret plan"); - } - }); + test("a resolver that cannot identify the caller is 401", async () => { + const res = await post( + appFor(() => null), + "/add", + { title: "t", text: "b" }, + ); + expect(res.status).toBe(401); + expect( + ((await res.json()) as { error: { code: string } }).error.code, + ).toBe("unauthorized"); + }); - test("forget, purge and retention-class work for the resolved creator only", async () => { - const owner = as("acme", "run"); - const other = as("acme", "other-run"); + test("a resolver that throws fails closed without leaking its message", async () => { + const before = await documentCount(); + const res = await post( + appFor(() => { + throw new Error("db connection string with secret=abc123"); + }), + "/add", + { title: "t", text: "b" }, + ); + expect(res.status).toBe(500); + expect(await res.text()).not.toContain("secret=abc123"); + expect(await documentCount()).toBe(before); + }); - const forgotten = await add(owner, "Forget via run"); - expect((await post(other, `/documents/${forgotten}/forget`)).status).toBe(403); - expect((await post(owner, `/documents/${forgotten}/forget`)).status).toBe(200); + test.each([ + ["empty", { tenantId: "", principalId: "" }], + ["whitespace", { tenantId: " ", principalId: "\t\n" }], + ] as const)( + "a resolver returning a %s identity is a 500, never a seated scope", + async (_label, resolved) => { + // The malformed principal holds an add grant, so a 500 proves the + // identity was rejected before the grant check. + const before = await documentCount(); + const res = await post( + appFor(() => resolved), + "/add", + { title: "t", text: "b" }, + ); + expect(await documentCount()).toBe(before); + expect(res.status).toBe(500); + expect( + ((await res.json()) as { error: { code: string } }).error.code, + ).toBe("invalid_resolved_caller"); + }, + ); - const purged = await add(owner, "Purge via run"); - expect((await post(owner, `/documents/${purged}/purge`)).status).toBe(200); + test("search, list and feed read only the resolved tenant, whatever the URL says", async () => { + const globexShare = await post(as("globex", "globex-user"), "/add", { + title: "Globex secret plan", + text: "globex secret plan body", + share: { tenant: true }, + }); + expect(globexShare.status).toBe(200); + await add(as("acme", "run"), "Acme run plan"); + const acme = as("acme", "run"); + + const search = await acme.request("/api/tenants/globex/memory/search", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ query: "plan" }), + }); + expect(search.status).toBe(200); + const { items } = (await search.json()) as { items: { title: string }[] }; + expect(items.map((i) => i.title)).toContain("Acme run plan"); + expect(items.map((i) => i.title)).not.toContain("Globex secret plan"); + + for (const route of ["list", "feed"]) { + const res = await acme.request(`/api/tenants/globex/memory/${route}`); + expect(res.status).toBe(200); + const body = JSON.stringify(await res.json()); + expect(body).toContain("Acme run plan"); + expect(body).not.toContain("Globex secret plan"); + } + }); - const classed = await add(owner, "Retention via run"); - const [version] = await db.sql<{ id: string }[]>` + test("forget, purge and retention-class work for the resolved creator only", async () => { + const owner = as("acme", "run"); + const other = as("acme", "other-run"); + + const forgotten = await add(owner, "Forget via run"); + expect((await post(other, `/documents/${forgotten}/forget`)).status).toBe( + 403, + ); + expect((await post(owner, `/documents/${forgotten}/forget`)).status).toBe( + 200, + ); + + const purged = await add(owner, "Purge via run"); + expect((await post(owner, `/documents/${purged}/purge`)).status).toBe( + 200, + ); + + const classed = await add(owner, "Retention via run"); + const [version] = await db.sql<{ id: string }[]>` SELECT id FROM memory.version WHERE document_id = ${classed}`; - const path = `/versions/${version?.id}/retention-class`; - expect((await post(other, path, { retention_class: "durable" })).status).toBe(403); - expect((await post(owner, path, { retention_class: "durable" })).status).toBe(200); - }); -}); + const path = `/versions/${version?.id}/retention-class`; + expect( + (await post(other, path, { retention_class: "durable" })).status, + ).toBe(403); + expect( + (await post(owner, path, { retention_class: "durable" })).status, + ).toBe(200); + }); + }, +); diff --git a/e2e/context-rows.test.ts b/e2e/context-rows.test.ts index 7ae30e9..2e28cf9 100644 --- a/e2e/context-rows.test.ts +++ b/e2e/context-rows.test.ts @@ -14,82 +14,93 @@ import { type TestDb, } from "./helpers.ts"; -describe.skipIf(testDatabaseUrl() === undefined)("synthesized context rows", () => { - let db: TestDb; - let memory: Memory | undefined; +describe.skipIf(testDatabaseUrl() === undefined)( + "synthesized context rows", + () => { + let db: TestDb; + let memory: Memory | undefined; - beforeAll(async () => { - db = await createTestDb(); - await seedPrincipal(db, "acme", "run"); - }); + beforeAll(async () => { + db = await createTestDb(); + await seedPrincipal(db, "acme", "run"); + }); - afterAll(async () => { - await memory?.close(); - await db?.close(); - }); + afterAll(async () => { + await memory?.close(); + await db?.close(); + }); - // The rows resolveCaller seats carry placeholder fields; this fails loudly - // if authorization or the routes ever read one. - test("requireGrant and the routes read only .id and .tenantId", async () => { - function canary(row: T, allowed: (keyof T)[]): T { - return new Proxy(row, { - get(target, prop, receiver) { - if (typeof prop === "string" && !allowed.includes(prop as keyof T)) { - throw new Error(`unexpected field access on a synthesized row: ${prop}`); - } - return Reflect.get(target, prop, receiver); - }, - }); - } - const grantStore = createInMemoryGrantStore([allow("run", "add")]); - memory = createTestMemory(db, grantStore); - const app = new Hono(); - app.use("*", async (c, next) => { - c.set( - "principal", - canary( - { - id: "run", - tenantId: "acme", - kind: "agent", - refId: "run", - status: "active", - createdAt: new Date(0), - updatedAt: new Date(0), - }, - ["id", "tenantId"], - ), - ); - c.set( - "tenant", - canary( - { - id: "acme", - name: "acme", - slug: "acme", - domain: "", - parentId: null, - config: null, - createdAt: new Date(0), - updatedAt: new Date(0), + // The rows resolveCaller seats carry placeholder fields; this fails loudly + // if authorization or the routes ever read one. + test("requireGrant and the routes read only .id and .tenantId", async () => { + function canary(row: T, allowed: (keyof T)[]): T { + return new Proxy(row, { + get(target, prop, receiver) { + if ( + typeof prop === "string" && + !allowed.includes(prop as keyof T) + ) { + throw new Error( + `unexpected field access on a synthesized row: ${prop}`, + ); + } + return Reflect.get(target, prop, receiver); }, - ["id"], - ), + }); + } + const grantStore = createInMemoryGrantStore([allow("run", "add")]); + memory = createTestMemory(db, grantStore); + const app = new Hono(); + app.use("*", async (c, next) => { + c.set( + "principal", + canary( + { + id: "run", + tenantId: "acme", + kind: "agent", + refId: "run", + status: "active", + createdAt: new Date(0), + updatedAt: new Date(0), + }, + ["id", "tenantId"], + ), + ); + c.set( + "tenant", + canary( + { + id: "acme", + name: "acme", + slug: "acme", + domain: "", + parentId: null, + config: null, + createdAt: new Date(0), + updatedAt: new Date(0), + }, + ["id"], + ), + ); + await next(); + }); + app.route( + "/api/tenants/:tenantId/memory", + createMemoryRoutes({ + memory, + requireGrant: createRequireGrant({ + grantStore, + conditionRegistry: {}, + }), + }), ); - await next(); - }); - app.route( - "/api/tenants/:tenantId/memory", - createMemoryRoutes({ - memory, - requireGrant: createRequireGrant({ grantStore, conditionRegistry: {} }), - }), - ); - const res = await app.request("/api/tenants/acme/memory/add", { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ title: "t", text: "body" }), + const res = await app.request("/api/tenants/acme/memory/add", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ title: "t", text: "body" }), + }); + expect(res.status).toBe(200); }); - expect(res.status).toBe(200); - }); -}); + }, +); diff --git a/e2e/embed-client.test.ts b/e2e/embed-client.test.ts index 8fb4826..e9efbc2 100644 --- a/e2e/embed-client.test.ts +++ b/e2e/embed-client.test.ts @@ -12,7 +12,11 @@ const stub = startHttpStub(); afterAll(() => stub.stop()); beforeEach(() => stub.reset()); -const openai = { baseUrl: stub.url, modelId: "text-embed-3", apiStyle: "openai" } as const; +const openai = { + baseUrl: stub.url, + modelId: "text-embed-3", + apiStyle: "openai", +} as const; describe("embedTexts", () => { test("sends nothing for empty input", async () => { @@ -26,35 +30,55 @@ describe("embedTexts", () => { const texts = body.input ?? body.inputs ?? []; const vectors = texts.map((t) => [t.length, 0]); if (req.path === "/v1/embeddings") { - return Response.json({ data: vectors.map((embedding) => ({ embedding })) }); + return Response.json({ + data: vectors.map((embedding) => ({ embedding })), + }); } if (req.path === "/embed") return Response.json(vectors); return Response.json({ embeddings: vectors }); }; - expect(await embedTexts(["a", "bb"], openai)).toEqual([[1, 0], [2, 0]]); + expect(await embedTexts(["a", "bb"], openai)).toEqual([ + [1, 0], + [2, 0], + ]); expect( - await embedTexts(["a", "bb"], { baseUrl: stub.url, modelId: "bge-m3", apiStyle: "tei" }), - ).toEqual([[1, 0], [2, 0]]); + await embedTexts(["a", "bb"], { + baseUrl: stub.url, + modelId: "bge-m3", + apiStyle: "tei", + }), + ).toEqual([ + [1, 0], + [2, 0], + ]); expect( await embedTexts(["a", "bb"], { baseUrl: stub.url, modelId: "nomic-embed-text", apiStyle: "ollama", }), - ).toEqual([[1, 0], [2, 0]]); + ).toEqual([ + [1, 0], + [2, 0], + ]); expect(stub.requests.map((r) => [r.path, r.body])).toEqual([ ["/v1/embeddings", { model: "text-embed-3", input: ["a", "bb"] }], ["/embed", { inputs: ["a", "bb"] }], - ["/api/embed", { model: "nomic-embed-text", input: ["a", "bb"], truncate: true }], + [ + "/api/embed", + { model: "nomic-embed-text", input: ["a", "bb"], truncate: true }, + ], ]); }); test("sends the bearer token only when apiKey is set, and batches per batchSize", async () => { stub.reply = (req) => Response.json({ - data: (req.body as { input: string[] }).input.map(() => ({ embedding: [1] })), + data: (req.body as { input: string[] }).input.map(() => ({ + embedding: [1], + })), }); await embedTexts(["a", "b", "c"], { @@ -67,7 +91,10 @@ describe("embedTexts", () => { await embedTexts(["d"], openai); expect( - stub.requests.map((r) => [r.authorization, (r.body as { input: string[] }).input]), + stub.requests.map((r) => [ + r.authorization, + (r.body as { input: string[] }).input, + ]), ).toEqual([ ["Bearer secret", ["a", "b"]], ["Bearer secret", ["c"]], @@ -94,13 +121,14 @@ describe("embedTexts", () => { apiStyle: "openai", timeoutMs: 20, }), - ).rejects.toThrow( - EmbedTimeoutError, - ); + ).rejects.toThrow(EmbedTimeoutError); }); }); test("probeEmbedDims returns the served vector length", async () => { - stub.reply = () => Response.json({ data: [{ embedding: new Array(768).fill(0) }] }); + stub.reply = () => + Response.json({ + data: [{ embedding: Array.from({ length: 768 }, () => 0) }], + }); expect(await probeEmbedDims(openai)).toBe(768); }); diff --git a/e2e/grants.test.ts b/e2e/grants.test.ts index bfdb095..63244dd 100644 --- a/e2e/grants.test.ts +++ b/e2e/grants.test.ts @@ -14,226 +14,272 @@ import { type TestDb, } from "./helpers.ts"; -describe.skipIf(testDatabaseUrl() === undefined)("grants, forget and purge", () => { - let db: TestDb; - let memory: Memory | undefined; - let app: Hono; - - beforeAll(async () => { - db = await createTestDb(); - for (const principal of ["alice", "carol", "dave", "erin", "frank"]) { - await seedPrincipal(db, "acme", principal); - } - const grantStore = createInMemoryGrantStore([ - ...["add", "search", "forget", "purge"].map((a) => allow("alice", a)), - ...["search", "forget", "purge"].map((a) => allow("carol", a)), - { - id: "g-carol-tenant-tag", - resource: "memory.tenant:acme", - action: "search", - effect: "allow", - origin: "role", - conditions: null, - expiresAt: null, - roleId: null, - principalId: "carol", - }, - allow("dave", "add"), - ...["search", "forget"].map((a) => allow("erin", a)), - ...["capture", "find"].map((a) => allow("frank", a)), - ]); - memory = createTestMemory(db, grantStore); - app = createTestApp({ - memory, - grantStore, - callers: { - alice: { tenantId: "acme", principalId: "alice" }, - carol: { tenantId: "acme", principalId: "carol" }, - dave: { tenantId: "acme", principalId: "dave" }, - erin: { tenantId: "acme", principalId: "erin" }, - frank: { tenantId: "acme", principalId: "frank" }, - }, +describe.skipIf(testDatabaseUrl() === undefined)( + "grants, forget and purge", + () => { + let db: TestDb; + let memory: Memory | undefined; + let app: Hono; + + beforeAll(async () => { + db = await createTestDb(); + for (const principal of ["alice", "carol", "dave", "erin", "frank"]) { + await seedPrincipal(db, "acme", principal); + } + const grantStore = createInMemoryGrantStore([ + ...["add", "search", "forget", "purge"].map((a) => allow("alice", a)), + ...["search", "forget", "purge"].map((a) => allow("carol", a)), + { + id: "g-carol-tenant-tag", + resource: "memory.tenant:acme", + action: "search", + effect: "allow", + origin: "role", + conditions: null, + expiresAt: null, + roleId: null, + principalId: "carol", + }, + allow("dave", "add"), + ...["search", "forget"].map((a) => allow("erin", a)), + ...["capture", "find"].map((a) => allow("frank", a)), + ]); + memory = createTestMemory(db, grantStore); + app = createTestApp({ + memory, + grantStore, + callers: { + alice: { tenantId: "acme", principalId: "alice" }, + carol: { tenantId: "acme", principalId: "carol" }, + dave: { tenantId: "acme", principalId: "dave" }, + erin: { tenantId: "acme", principalId: "erin" }, + frank: { tenantId: "acme", principalId: "frank" }, + }, + }); }); - }); - - afterAll(async () => { - await memory?.close(); - await db?.close(); - }); - - function post(token: string, path: string, body: unknown = {}) { - return app.request(`/api/tenants/acme/memory${path}`, { - method: "POST", - headers: { - authorization: `Bearer ${token}`, - "content-type": "application/json", - }, - body: JSON.stringify(body), + + afterAll(async () => { + await memory?.close(); + await db?.close(); }); - } - async function addDocument(title: string): Promise { - const res = await post("alice", "/add", { title, text: `${title} body` }); - expect(res.status).toBe(200); - return ((await res.json()) as { documentId: string }).documentId; - } + function post(token: string, path: string, body: unknown = {}) { + return app.request(`/api/tenants/acme/memory${path}`, { + method: "POST", + headers: { + authorization: `Bearer ${token}`, + "content-type": "application/json", + }, + body: JSON.stringify(body), + }); + } + + async function addDocument(title: string): Promise { + const res = await post("alice", "/add", { title, text: `${title} body` }); + expect(res.status).toBe(200); + return ((await res.json()) as { documentId: string }).documentId; + } - test("search without the memory:search grant is 403", async () => { - const res = await post("dave", "/search", { query: "anything" }); - expect(res.status).toBe(403); - }); + test("search without the memory:search grant is 403", async () => { + const res = await post("dave", "/search", { query: "anything" }); + expect(res.status).toBe(403); + }); - test("forget succeeds for the creator and is 403 for anyone else", async () => { - const documentId = await addDocument("Forget me"); + test("forget succeeds for the creator and is 403 for anyone else", async () => { + const documentId = await addDocument("Forget me"); - const other = await post("carol", `/documents/${documentId}/forget`); - expect(other.status).toBe(403); + const other = await post("carol", `/documents/${documentId}/forget`); + expect(other.status).toBe(403); - const creator = await post("alice", `/documents/${documentId}/forget`); - expect(creator.status).toBe(200); - const [row] = await db.sql<{ status: string }[]>` + const creator = await post("alice", `/documents/${documentId}/forget`); + expect(creator.status).toBe(200); + const [row] = await db.sql<{ status: string }[]>` SELECT status FROM memory.version WHERE document_id = ${documentId}`; - expect(row?.status).toBe("tombstoned"); - }); + expect(row?.status).toBe("tombstoned"); + }); - test("purge removes the document, its versions and its chunks", async () => { - const documentId = await addDocument("Purge me"); - const [before] = await db.sql<{ n: number }[]>` + test("purge removes the document, its versions and its chunks", async () => { + const documentId = await addDocument("Purge me"); + const [before] = await db.sql<{ n: number }[]>` SELECT count(*)::int AS n FROM memory.chunk c JOIN memory.version v ON v.id = c.version_id WHERE v.document_id = ${documentId}`; - expect(before?.n).toBeGreaterThan(0); + expect(before?.n).toBeGreaterThan(0); - const res = await post("alice", `/documents/${documentId}/purge`); - expect(res.status).toBe(200); - expect(await res.json()).toMatchObject({ documentId, deleted: true }); + const res = await post("alice", `/documents/${documentId}/purge`); + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ documentId, deleted: true }); - const [left] = await db.sql<{ docs: number; versions: number; chunks: number }[]>` + const [left] = await db.sql< + { docs: number; versions: number; chunks: number }[] + >` SELECT (SELECT count(*)::int FROM memory.document WHERE id = ${documentId}) AS docs, (SELECT count(*)::int FROM memory.version WHERE document_id = ${documentId}) AS versions, (SELECT count(*)::int FROM memory.chunk WHERE document_id = ${documentId}) AS chunks`; - expect(left).toEqual({ docs: 0, versions: 0, chunks: 0 }); - }); - - test("legacy capture and find grants authorize neither add nor search", async () => { - expect((await post("frank", "/add", { title: "t", text: "b" })).status).toBe(403); - expect((await post("frank", "/search", { query: "q" })).status).toBe(403); - }); - - test("a request without a session is 401", async () => { - const res = await app.request("/api/tenants/acme/memory/search", { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ query: "q" }), + expect(left).toEqual({ docs: 0, versions: 0, chunks: 0 }); + }); + + test("legacy capture and find grants authorize neither add nor search", async () => { + expect( + (await post("frank", "/add", { title: "t", text: "b" })).status, + ).toBe(403); + expect((await post("frank", "/search", { query: "q" })).status).toBe(403); + }); + + test("a request without a session is 401", async () => { + const res = await app.request("/api/tenants/acme/memory/search", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ query: "q" }), + }); + expect(res.status).toBe(401); }); - expect(res.status).toBe(401); - }); - - test("malformed bodies and out-of-range limits are 400", async () => { - expect((await post("alice", "/add", { title: "no text" })).status).toBe(400); - expect((await post("alice", "/search", { query: "q", limit: 51 })).status).toBe(400); - expect((await post("alice", "/search", { query: "q", kinds: [1] })).status).toBe(400); - }); - - test("list and search show a document to its creator and to holders of a grant on its tags only", async () => { - await post("alice", "/add", { title: "Alice private", text: "only alice" }); - await post("alice", "/add", { - title: "Shared with carol", - text: "shared with the tenant", - share: { tenant: true }, + + test("malformed bodies and out-of-range limits are 400", async () => { + expect((await post("alice", "/add", { title: "no text" })).status).toBe( + 400, + ); + expect( + (await post("alice", "/search", { query: "q", limit: 51 })).status, + ).toBe(400); + expect( + (await post("alice", "/search", { query: "q", kinds: [1] })).status, + ).toBe(400); }); - const titles = async (token: string) => { + + test("list and search show a document to its creator and to holders of a grant on its tags only", async () => { + await post("alice", "/add", { + title: "Alice private", + text: "only alice", + }); + await post("alice", "/add", { + title: "Shared with carol", + text: "shared with the tenant", + share: { tenant: true }, + }); + const titles = async (token: string) => { + const res = await app.request("/api/tenants/acme/memory/list", { + headers: { authorization: `Bearer ${token}` }, + }); + expect(res.status).toBe(200); + return JSON.stringify(await res.json()); + }; + const forAlice = await titles("alice"); + const forCarol = await titles("carol"); + const forErin = await titles("erin"); + expect(forAlice).toContain("Alice private"); + expect(forCarol).toContain("Shared with carol"); + expect(forCarol).not.toContain("Alice private"); + expect(forErin).not.toContain("Alice private"); + expect(forErin).not.toContain("Shared with carol"); + + const searched = async (token: string) => { + const res = await post(token, "/search", { query: "only alice" }); + expect(res.status).toBe(200); + const { items } = (await res.json()) as { items: { title: string }[] }; + return items.map((i) => i.title); + }; + expect(await searched("alice")).toContain("Alice private"); + expect(await searched("carol")).not.toContain("Alice private"); + expect(await searched("erin")).not.toContain("Alice private"); + }); + + test("purge is refused for a non-creator holding the purge grant", async () => { + const documentId = await addDocument("Carol cannot purge"); + expect( + (await post("carol", `/documents/${documentId}/purge`)).status, + ).toBe(403); + }); + + test("list needs the search grant", async () => { const res = await app.request("/api/tenants/acme/memory/list", { - headers: { authorization: `Bearer ${token}` }, + headers: { authorization: "Bearer dave" }, }); - expect(res.status).toBe(200); - return JSON.stringify(await res.json()); - }; - const forAlice = await titles("alice"); - const forCarol = await titles("carol"); - const forErin = await titles("erin"); - expect(forAlice).toContain("Alice private"); - expect(forCarol).toContain("Shared with carol"); - expect(forCarol).not.toContain("Alice private"); - expect(forErin).not.toContain("Alice private"); - expect(forErin).not.toContain("Shared with carol"); - - const searched = async (token: string) => { - const res = await post(token, "/search", { query: "only alice" }); - expect(res.status).toBe(200); - const { items } = (await res.json()) as { items: { title: string }[] }; - return items.map((i) => i.title); - }; - expect(await searched("alice")).toContain("Alice private"); - expect(await searched("carol")).not.toContain("Alice private"); - expect(await searched("erin")).not.toContain("Alice private"); - }); - - test("purge is refused for a non-creator holding the purge grant", async () => { - const documentId = await addDocument("Carol cannot purge"); - expect((await post("carol", `/documents/${documentId}/purge`)).status).toBe(403); - }); - - test("list needs the search grant", async () => { - const res = await app.request("/api/tenants/acme/memory/list", { - headers: { authorization: "Bearer dave" }, + expect(res.status).toBe(403); }); - expect(res.status).toBe(403); - }); - - test("search passes kinds and entity_ids through, and empty arrays filter nothing", async () => { - await post("alice", "/add", { title: "Mango decision", text: "Mango wins.", kind: "decision" }); - await post("alice", "/add", { title: "Mango note", text: "Mango is ripe." }); - const titles = async (body: Record) => { - const res = await post("alice", "/search", body); - expect(res.status).toBe(200); - const { items } = (await res.json()) as { items: { title: string }[] }; - return items.map((i) => i.title).sort(); - }; - expect(await titles({ query: "mango", kinds: ["decision"] })).toEqual(["Mango decision"]); - expect(await titles({ query: "mango", kinds: [], entity_ids: [] })).toEqual([ - "Mango decision", - "Mango note", - ]); - expect(await titles({ query: "mango", entity_ids: ["entity-none"] })).toEqual([]); - }); - - test("the forget grant does not authorize purge, and forget never deletes rows", async () => { - const documentId = await addDocument("Erin cannot purge"); - await post("alice", `/documents/${documentId}/forget`); - expect((await post("erin", `/documents/${documentId}/purge`)).status).toBe(403); - const [row] = await db.sql<{ n: number }[]>` + + test("search passes kinds and entity_ids through, and empty arrays filter nothing", async () => { + await post("alice", "/add", { + title: "Mango decision", + text: "Mango wins.", + kind: "decision", + }); + await post("alice", "/add", { + title: "Mango note", + text: "Mango is ripe.", + }); + const titles = async (body: Record) => { + const res = await post("alice", "/search", body); + expect(res.status).toBe(200); + const { items } = (await res.json()) as { items: { title: string }[] }; + return items.map((i) => i.title).sort(); + }; + expect(await titles({ query: "mango", kinds: ["decision"] })).toEqual([ + "Mango decision", + ]); + expect( + await titles({ query: "mango", kinds: [], entity_ids: [] }), + ).toEqual(["Mango decision", "Mango note"]); + expect( + await titles({ query: "mango", entity_ids: ["entity-none"] }), + ).toEqual([]); + }); + + test("the forget grant does not authorize purge, and forget never deletes rows", async () => { + const documentId = await addDocument("Erin cannot purge"); + await post("alice", `/documents/${documentId}/forget`); + expect( + (await post("erin", `/documents/${documentId}/purge`)).status, + ).toBe(403); + const [row] = await db.sql<{ n: number }[]>` SELECT count(*)::int AS n FROM memory.document WHERE id = ${documentId}`; - expect(row?.n).toBe(1); - }); - - test("unknown documents and versions are 404, and whitespace ids are 400", async () => { - expect((await post("alice", "/documents/doc-missing/forget")).status).toBe(404); - expect((await post("alice", "/documents/doc-missing/purge")).status).toBe(404); - expect( - (await post("alice", "/versions/ver-missing/retention-class", { retention_class: "durable" })) - .status, - ).toBe(404); - expect((await post("alice", "/documents/%20/forget")).status).toBe(400); - expect((await post("alice", "/documents/%20/purge")).status).toBe(400); - expect( - (await post("alice", "/versions/%20/retention-class", { retention_class: "durable" })) - .status, - ).toBe(400); - }); - - test("retention-class changes a version for its creator only and validates the class", async () => { - const documentId = await addDocument("Retention target"); - const [version] = await db.sql<{ id: string }[]>` + expect(row?.n).toBe(1); + }); + + test("unknown documents and versions are 404, and whitespace ids are 400", async () => { + expect( + (await post("alice", "/documents/doc-missing/forget")).status, + ).toBe(404); + expect((await post("alice", "/documents/doc-missing/purge")).status).toBe( + 404, + ); + expect( + ( + await post("alice", "/versions/ver-missing/retention-class", { + retention_class: "durable", + }) + ).status, + ).toBe(404); + expect((await post("alice", "/documents/%20/forget")).status).toBe(400); + expect((await post("alice", "/documents/%20/purge")).status).toBe(400); + expect( + ( + await post("alice", "/versions/%20/retention-class", { + retention_class: "durable", + }) + ).status, + ).toBe(400); + }); + + test("retention-class changes a version for its creator only and validates the class", async () => { + const documentId = await addDocument("Retention target"); + const [version] = await db.sql<{ id: string }[]>` SELECT id FROM memory.version WHERE document_id = ${documentId}`; - const path = `/versions/${version?.id}/retention-class`; + const path = `/versions/${version?.id}/retention-class`; - expect((await post("alice", path, { retention_class: "forever" })).status).toBe(400); - expect((await post("carol", path, { retention_class: "durable" })).status).toBe(403); - expect((await post("alice", path, { retention_class: "durable" })).status).toBe(200); - const [row] = await db.sql<{ retention_class: string }[]>` + expect( + (await post("alice", path, { retention_class: "forever" })).status, + ).toBe(400); + expect( + (await post("carol", path, { retention_class: "durable" })).status, + ).toBe(403); + expect( + (await post("alice", path, { retention_class: "durable" })).status, + ).toBe(200); + const [row] = await db.sql<{ retention_class: string }[]>` SELECT retention_class FROM memory.version WHERE id = ${version?.id ?? ""}`; - expect(row?.retention_class).toBe("durable"); - }); -}); + expect(row?.retention_class).toBe("durable"); + }); + }, +); diff --git a/e2e/helpers.ts b/e2e/helpers.ts index ca0aa62..53d0f41 100644 --- a/e2e/helpers.ts +++ b/e2e/helpers.ts @@ -72,7 +72,9 @@ export async function createEmptyDb(): Promise { try { await sql.end(); } finally { - await admin.unsafe(`DROP DATABASE IF EXISTS "${database}" WITH (FORCE)`); + await admin.unsafe( + `DROP DATABASE IF EXISTS "${database}" WITH (FORCE)`, + ); await admin.end(); } }, diff --git a/e2e/migrations.test.ts b/e2e/migrations.test.ts index 4424fe9..804aaa6 100644 --- a/e2e/migrations.test.ts +++ b/e2e/migrations.test.ts @@ -44,7 +44,9 @@ describe.skipIf(testDatabaseUrl() === undefined)("memory migrations", () => { await runMemoryMigrations(db.config, options); const afterSecond = await snapshot("memory"); - expect(afterFirst.some((i) => i.startsWith("column document.id "))).toBe(true); + expect(afterFirst.some((i) => i.startsWith("column document.id "))).toBe( + true, + ); expect(afterSecond).toEqual(afterFirst); expect(await snapshot("public")).toEqual(publicBefore); }); @@ -73,106 +75,118 @@ describe.skipIf(testDatabaseUrl() === undefined)("memory migrations", () => { }, 30_000); }); -describe.skipIf(testDatabaseUrl() === undefined)("concurrent memory migrations", () => { - let db: TestDb; +describe.skipIf(testDatabaseUrl() === undefined)( + "concurrent memory migrations", + () => { + let db: TestDb; - beforeAll(async () => { - db = await createEmptyDb(); - await runMigrations(db.config, { schema: "public" }); - }); + beforeAll(async () => { + db = await createEmptyDb(); + await runMigrations(db.config, { schema: "public" }); + }); - afterAll(async () => { - await db.close(); - }); + afterAll(async () => { + await db.close(); + }); - // Every replica of a rolling deploy runs the migrations at boot. - const runThree = () => - Promise.all([0, 1, 2].map(() => runMemoryMigrations(db.config, options))); + // Every replica of a rolling deploy runs the migrations at boot. + const runThree = () => + Promise.all([0, 1, 2].map(() => runMemoryMigrations(db.config, options))); - test("three runners on a fresh database all succeed", async () => { - await runThree(); - }, 30_000); + test("three runners on a fresh database all succeed", async () => { + await runThree(); + }, 30_000); - test("three runners on a migrated database all succeed", async () => { - await runThree(); - }, 30_000); -}); + test("three runners on a migrated database all succeed", async () => { + await runThree(); + }, 30_000); + }, +); -describe.skipIf(testDatabaseUrl() === undefined)("memory migrations in a non-public host schema", () => { - let db: TestDb; +describe.skipIf(testDatabaseUrl() === undefined)( + "memory migrations in a non-public host schema", + () => { + let db: TestDb; - beforeAll(async () => { - db = await createEmptyDb(); - await runMigrations(db.config, { schema: "hub" }); - }); + beforeAll(async () => { + db = await createEmptyDb(); + await runMigrations(db.config, { schema: "hub" }); + }); - afterAll(async () => { - await db.close(); - }); + afterAll(async () => { + await db.close(); + }); - test("foreign keys point at the host schema, and a replay against another schema fails", async () => { - const hub = { schema: "hub", ftsLanguage: "english" }; - await runMemoryMigrations(db.config, hub); - await runMemoryMigrations(db.config, hub); + test("foreign keys point at the host schema, and a replay against another schema fails", async () => { + const hub = { schema: "hub", ftsLanguage: "english" }; + await runMemoryMigrations(db.config, hub); + await runMemoryMigrations(db.config, hub); - const targets = await db.sql<{ target: string }[]>` + const targets = await db.sql<{ target: string }[]>` SELECT DISTINCT confrelid::regclass::text AS target FROM pg_constraint c JOIN pg_namespace n ON n.oid = c.connamespace WHERE n.nspname = 'memory' AND c.contype = 'f' AND confrelid::regclass::text NOT LIKE 'memory.%' ORDER BY 1`; - expect(targets.map((t) => t.target)).toEqual(["hub.principal", "hub.tenant"]); - - await runMigrations(db.config, { schema: "public" }); - await expect(runMemoryMigrations(db.config, options)).rejects.toThrow( - "already exists", - ); - }); -}); - -describe.skipIf(testDatabaseUrl() === undefined)("upgrading a database the ledger runner left before 0004", () => { - let db: TestDb; - - beforeAll(async () => { - db = await createEmptyDb(); - await runMigrations(db.config, { schema: "public" }); - const dir = join(import.meta.dirname, "..", "migrations"); - for (const file of (await readdir(dir)).sort()) { - if (file >= "0004") break; - const raw = await readFile(join(dir, file), "utf8"); - await db.sql.unsafe(raw.replaceAll("{{FTS_LANGUAGE}}", "english")); - } - await db.sql` + expect(targets.map((t) => t.target)).toEqual([ + "hub.principal", + "hub.tenant", + ]); + + await runMigrations(db.config, { schema: "public" }); + await expect(runMemoryMigrations(db.config, options)).rejects.toThrow( + "already exists", + ); + }); + }, +); + +describe.skipIf(testDatabaseUrl() === undefined)( + "upgrading a database the ledger runner left before 0004", + () => { + let db: TestDb; + + beforeAll(async () => { + db = await createEmptyDb(); + await runMigrations(db.config, { schema: "public" }); + const dir = join(import.meta.dirname, "..", "migrations"); + for (const file of (await readdir(dir)).sort()) { + if (file >= "0004") break; + const raw = await readFile(join(dir, file), "utf8"); + await db.sql.unsafe(raw.replaceAll("{{FTS_LANGUAGE}}", "english")); + } + await db.sql` INSERT INTO memory.document (id, tenant_id, kind, title, adapter, external_ref) VALUES ('doc-old', 'acme', 'note', 'Old claim', 'test', 'old')`; - await db.sql` + await db.sql` INSERT INTO memory.version (id, tenant_id, document_id, version, content_hash, occurred_at, created_by_kind, provenance) VALUES ('ver-old', 'acme', 'doc-old', 1, 'h', now(), 'agent', 'inferred')`; - await db.sql` + await db.sql` INSERT INTO public.tenant (id, name, slug, domain) VALUES ('acme', 'acme', 'acme', 'acme.test')`; - }); + }); - afterAll(async () => { - await db.close(); - }); + afterAll(async () => { + await db.close(); + }); - test("the temporal_class backfill applies once and a later replay leaves new rows alone", async () => { - await runMemoryMigrations(db.config, options); - const [old] = await db.sql<{ temporal_class: string }[]>` + test("the temporal_class backfill applies once and a later replay leaves new rows alone", async () => { + await runMemoryMigrations(db.config, options); + const [old] = await db.sql<{ temporal_class: string }[]>` SELECT temporal_class FROM memory.version WHERE id = 'ver-old'`; - expect(old?.temporal_class).toBe("state"); + expect(old?.temporal_class).toBe("state"); - await db.sql` + await db.sql` INSERT INTO memory.version (id, tenant_id, document_id, version, content_hash, occurred_at, created_by_kind, provenance, temporal_class) VALUES ('ver-new', 'acme', 'doc-old', 2, 'h2', now(), 'agent', 'inferred', 'event')`; - await runMemoryMigrations(db.config, options); - const [fresh] = await db.sql<{ temporal_class: string }[]>` + await runMemoryMigrations(db.config, options); + const [fresh] = await db.sql<{ temporal_class: string }[]>` SELECT temporal_class FROM memory.version WHERE id = 'ver-new'`; - expect(fresh?.temporal_class).toBe("event"); - }); -}); + expect(fresh?.temporal_class).toBe("event"); + }); + }, +); diff --git a/e2e/plane.test.ts b/e2e/plane.test.ts index aaeb617..6a98f70 100644 --- a/e2e/plane.test.ts +++ b/e2e/plane.test.ts @@ -59,21 +59,48 @@ describe.skipIf(testDatabaseUrl() === undefined)("in-process memory", () => { } test.each([ - ["search below 1", (m: Memory) => - m.search({ tenantId: "acme", principalId: "alice", query: "q", limit: 0 })], - ["search above 50", (m: Memory) => - m.search({ tenantId: "acme", principalId: "alice", query: "q", limit: 51 })], - ["list below 1", (m: Memory) => - m.list({ tenantId: "acme", principalId: "alice", limit: 0 })], - ["list above 100", (m: Memory) => - m.list({ tenantId: "acme", principalId: "alice", limit: 101 })], + [ + "search below 1", + (m: Memory) => + m.search({ + tenantId: "acme", + principalId: "alice", + query: "q", + limit: 0, + }), + ], + [ + "search above 50", + (m: Memory) => + m.search({ + tenantId: "acme", + principalId: "alice", + query: "q", + limit: 51, + }), + ], + [ + "list below 1", + (m: Memory) => + m.list({ tenantId: "acme", principalId: "alice", limit: 0 }), + ], + [ + "list above 100", + (m: Memory) => + m.list({ tenantId: "acme", principalId: "alice", limit: 101 }), + ], ] as const)("rejects a limit %s with a 400", async (_label, call) => { const err = await rejection(call(memory as Memory)); expect(err.status).toBe(400); }); test("add needs exactly one of content or file", async () => { - const neither = await rejection((memory as Memory).add({ tenantId: "acme", principalId: "alice" } as MemoryAddParams)); + const neither = await rejection( + (memory as Memory).add({ + tenantId: "acme", + principalId: "alice", + } as MemoryAddParams), + ); const both = await rejection( (memory as Memory).add({ tenantId: "acme", @@ -105,7 +132,10 @@ describe.skipIf(testDatabaseUrl() === undefined)("in-process memory", () => { SELECT d.title, c.text FROM memory.document d JOIN memory.chunk c ON c.document_id = d.id WHERE d.id = ${documentId}`; - expect(row).toEqual({ title: "From extractor", text: "extracted from note.pdf" }); + expect(row).toEqual({ + title: "From extractor", + text: "extracted from note.pdf", + }); }); test("share maps to access tags, peers also get the document tag, and the default is owner-only", async () => { diff --git a/e2e/rerank-client.test.ts b/e2e/rerank-client.test.ts index 7dd7827..e5faf50 100644 --- a/e2e/rerank-client.test.ts +++ b/e2e/rerank-client.test.ts @@ -36,7 +36,9 @@ const docs = [ ]; function teiTexts(): string[] { - return stub.requests.map((r) => (r.body as { texts: string[] }).texts[0] ?? ""); + return stub.requests.map( + (r) => (r.body as { texts: string[] }).texts[0] ?? "", + ); } describe("rerankDocuments", () => { @@ -54,7 +56,9 @@ describe("rerankDocuments", () => { ]); } if (req.path === "/v2/rerank") { - return Response.json({ results: [{ index: 0, relevance_score: 0.75 }] }); + return Response.json({ + results: [{ index: 0, relevance_score: 0.75 }], + }); } return Response.json({ data: [ @@ -76,27 +80,38 @@ describe("rerankDocuments", () => { }), ).toEqual([{ id: "chunk-a", score: 0.75 }]); expect( - await rerankDocuments("my query", docs, { baseUrl: stub.url, apiStyle: "voyage" }), + await rerankDocuments("my query", docs, { + baseUrl: stub.url, + apiStyle: "voyage", + }), ).toEqual([ { id: "chunk-b", score: 0.6 }, { id: "chunk-a", score: 0.4 }, ]); const texts = ["alpha content", "beta content"]; - expect(stub.requests.map((r) => [r.path, r.authorization, r.body])).toEqual([ - ["/rerank", null, { query: "my query", texts }], + expect(stub.requests.map((r) => [r.path, r.authorization, r.body])).toEqual( [ - "/v2/rerank", - "Bearer secret-key", - { model: DEFAULT_RERANK_MODEL, query: "my query", documents: texts }, + ["/rerank", null, { query: "my query", texts }], + [ + "/v2/rerank", + "Bearer secret-key", + { model: DEFAULT_RERANK_MODEL, query: "my query", documents: texts }, + ], + [ + "/v1/rerank", + null, + { model: DEFAULT_RERANK_MODEL, query: "my query", documents: texts }, + ], ], - ["/v1/rerank", null, { model: DEFAULT_RERANK_MODEL, query: "my query", documents: texts }], - ]); + ); }); test("rejects a non-2xx reply with RerankHttpError", async () => { stub.reply = () => Response.json({ error: "boom" }, { status: 500 }); - await expect(rerankDocuments("q", docs, tei)).rejects.toBeInstanceOf(RerankHttpError); + await expect(rerankDocuments("q", docs, tei)).rejects.toBeInstanceOf( + RerankHttpError, + ); }); test("rejects a reply slower than timeoutMs with RerankTimeoutError", async () => { @@ -105,23 +120,70 @@ describe("rerankDocuments", () => { return Response.json([]); }; await expect( - rerankDocuments("q", docs, { baseUrl: stub.url, apiStyle: "tei", timeoutMs: 20 }), + rerankDocuments("q", docs, { + baseUrl: stub.url, + apiStyle: "tei", + timeoutMs: 20, + }), ).rejects.toBeInstanceOf(RerankTimeoutError); }); const baseBudget = defaultMaxDocCharsForModel("bge-reranker-base"); test.each([ - ["the default model's own budget", "", DEFAULT_MAX_DOC_CHARS + 500, tei, DEFAULT_MAX_DOC_CHARS], - ["an explicit bge-reranker-base budget", "", baseBudget + 500, teiBaseModel, baseBudget], - ["a configured maxDocChars, less a one-character query", "q", 1_200, teiBudget(1_000), 999], - ["an at-budget document, untouched", "", DEFAULT_MAX_DOC_CHARS, tei, DEFAULT_MAX_DOC_CHARS], - ["a configured maxDocChars, less a long query", "q".repeat(150), 1_200, teiBudget(1_000), 850], - ["exactly the minimum document budget", "q".repeat(800), 500, teiBudget(1_000), 200], - ] as const)("truncates to %s", async (_label, query, docChars, config, expected) => { - stub.reply = () => Response.json([{ index: 0, score: 0.5 }]); - await rerankDocuments(query, [{ id: "c", text: "x".repeat(docChars) }], config); - expect(teiTexts()[0]?.length).toBe(expected); - }); + [ + "the default model's own budget", + "", + DEFAULT_MAX_DOC_CHARS + 500, + tei, + DEFAULT_MAX_DOC_CHARS, + ], + [ + "an explicit bge-reranker-base budget", + "", + baseBudget + 500, + teiBaseModel, + baseBudget, + ], + [ + "a configured maxDocChars, less a one-character query", + "q", + 1_200, + teiBudget(1_000), + 999, + ], + [ + "an at-budget document, untouched", + "", + DEFAULT_MAX_DOC_CHARS, + tei, + DEFAULT_MAX_DOC_CHARS, + ], + [ + "a configured maxDocChars, less a long query", + "q".repeat(150), + 1_200, + teiBudget(1_000), + 850, + ], + [ + "exactly the minimum document budget", + "q".repeat(800), + 500, + teiBudget(1_000), + 200, + ], + ] as const)( + "truncates to %s", + async (_label, query, docChars, config, expected) => { + stub.reply = () => Response.json([{ index: 0, score: 0.5 }]); + await rerankDocuments( + query, + [{ id: "c", text: "x".repeat(docChars) }], + config, + ); + expect(teiTexts()[0]?.length).toBe(expected); + }, + ); test("refuses without a request when the query leaves too little document budget", async () => { const one = [{ id: "c", text: "x".repeat(500) }]; @@ -195,7 +257,9 @@ describe("validateRerankConfig", () => { validateRerankConfig({ baseUrl: "https://tei.example.com", apiStyle: "tei", - maxDocChars: (KNOWN_TEI_RERANK_MODEL_TOKEN_LIMITS[DEFAULT_RERANK_MODEL] ?? 0) * 100, + maxDocChars: + (KNOWN_TEI_RERANK_MODEL_TOKEN_LIMITS[DEFAULT_RERANK_MODEL] ?? 0) * + 100, }), ).toThrow(RerankConfigError); }); diff --git a/e2e/upgrade-from-0.1.0.test.ts b/e2e/upgrade-from-0.1.0.test.ts index cfcb21d..dbeb6bb 100644 --- a/e2e/upgrade-from-0.1.0.test.ts +++ b/e2e/upgrade-from-0.1.0.test.ts @@ -17,102 +17,110 @@ import { } from "./helpers.ts"; const DOCS = [ - { title: "Staging deploys", text: "Staging deploys run from main after every merge." }, + { + title: "Staging deploys", + text: "Staging deploys run from main after every merge.", + }, { title: "Vacation policy", text: "Request vacation two weeks ahead." }, ]; /** Every row of every memory table, keyed by table. */ type Snapshot = Record; -describe.skipIf(testDatabaseUrl() === undefined)("upgrading a 0.1.0 database", () => { - let db: TestDb; - let before: Snapshot; - const grantStore = createInMemoryGrantStore([ - allow("alice", "add"), - allow("alice", "search"), - ]); +describe.skipIf(testDatabaseUrl() === undefined)( + "upgrading a 0.1.0 database", + () => { + let db: TestDb; + let before: Snapshot; + const grantStore = createInMemoryGrantStore([ + allow("alice", "add"), + allow("alice", "search"), + ]); - async function snapshot(): Promise { - const tables = await db.sql<{ name: string }[]>` + async function snapshot(): Promise { + const tables = await db.sql<{ name: string }[]>` SELECT table_name AS name FROM information_schema.tables WHERE table_schema = 'memory' AND table_type = 'BASE TABLE' AND table_name <> '_migrations' ORDER BY 1`; - const result: Snapshot = {}; - for (const { name } of tables) { - const rows = await db.sql<{ row: string }[]>` + const result: Snapshot = {}; + for (const { name } of tables) { + const rows = await db.sql<{ row: string }[]>` SELECT to_jsonb(t)::text AS row FROM ${db.sql("memory")}.${db.sql(name)} t ORDER BY 1`; - result[name] = rows.map((r) => r.row); + result[name] = rows.map((r) => r.row); + } + return result; } - return result; - } - beforeAll(async () => { - db = await createEmptyDb(); - await runMigrations(db.config, { schema: "public" }); - await v010.runMemoryMigrations(db.databaseUrl, { ftsLanguage: "english" }); - await seedPrincipal(db, "acme", "alice"); + beforeAll(async () => { + db = await createEmptyDb(); + await runMigrations(db.config, { schema: "public" }); + await v010.runMemoryMigrations(db.databaseUrl, { + ftsLanguage: "english", + }); + await seedPrincipal(db, "acme", "alice"); - const legacy = v010.createMemory({ - config: testMemoryConfig(db), - grantStore, - conditionRegistry: {}, - }); - try { - for (const content of DOCS) { - await legacy.add({ tenantId: "acme", principalId: "alice", content }); + const legacy = v010.createMemory({ + config: testMemoryConfig(db), + grantStore, + conditionRegistry: {}, + }); + try { + for (const content of DOCS) { + await legacy.add({ tenantId: "acme", principalId: "alice", content }); + } + } finally { + await legacy.close(); } - } finally { - await legacy.close(); - } - // A distilled claim 0.1.0 already classed as an event: the 0004 - // backfill must not reclassify it on upgrade. - await db.sql` + // A distilled claim 0.1.0 already classed as an event: the 0004 + // backfill must not reclassify it on upgrade. + await db.sql` INSERT INTO memory.document (id, tenant_id, kind, title, adapter, external_ref) VALUES ('doc-claim', 'acme', 'claim', 'Deploy cadence', 'distiller', 'claim-1')`; - await db.sql` + await db.sql` INSERT INTO memory.version (id, tenant_id, document_id, version, content_hash, occurred_at, created_by_kind, provenance, temporal_class) VALUES ('ver-claim', 'acme', 'doc-claim', 1, 'claim', now(), 'agent', 'inferred', 'event')`; - await db.sql` + await db.sql` INSERT INTO memory.transform_config (id, tenant_id, name, version) VALUES ('tc-1', 'acme', 'chunker', 1)`; - await db.sql` + await db.sql` INSERT INTO memory.transform_run (id, tenant_id, config_id, generation, status) VALUES ('tr-1', 'acme', 'tc-1', 'gen-1', 'completed')`; - before = await snapshot(); + before = await snapshot(); - const options = { schema: "public", ftsLanguage: "english" }; - await runMemoryMigrations(db.config, options); - await runMemoryMigrations(db.config, options); - }); + const options = { schema: "public", ftsLanguage: "english" }; + await runMemoryMigrations(db.config, options); + await runMemoryMigrations(db.config, options); + }); - afterAll(async () => { - await db?.close(); - }); + afterAll(async () => { + await db?.close(); + }); - test("keeps every row and drops the 0.1.0 migration ledger", async () => { - expect(before["document"]).toHaveLength(DOCS.length + 1); - expect(before["raw_capture"]?.length).toBeGreaterThan(0); - expect(await snapshot()).toEqual(before); - const [ledger] = await db.sql<{ name: string | null }[]>` + test("keeps every row and drops the 0.1.0 migration ledger", async () => { + expect(before["document"]).toHaveLength(DOCS.length + 1); + expect(before["raw_capture"]?.length).toBeGreaterThan(0); + expect(await snapshot()).toEqual(before); + const [ledger] = await db.sql<{ name: string | null }[]>` SELECT to_regclass('memory._migrations')::text AS name`; - expect(ledger?.name).toBeNull(); - }); + expect(ledger?.name).toBeNull(); + }); - test("finds 0.1.0 documents through this version's search", async () => { - const memory = createTestMemory(db, grantStore); - try { - const { items } = await memory.search({ - tenantId: "acme", - principalId: "alice", - query: "staging deploys", - }); - expect(items[0]?.title).toBe("Staging deploys"); - } finally { - await memory.close(); - } - }); -}); + test("finds 0.1.0 documents through this version's search", async () => { + const memory = createTestMemory(db, grantStore); + try { + const { items } = await memory.search({ + tenantId: "acme", + principalId: "alice", + query: "staging deploys", + }); + expect(items[0]?.title).toBe("Staging deploys"); + } finally { + await memory.close(); + } + }); + }, +); diff --git a/package.json b/package.json index 89ee07e..85ed83c 100644 --- a/package.json +++ b/package.json @@ -2,6 +2,32 @@ "name": "@corbits/memory", "version": "0.2.0", "description": "Mountable memory add/search/list SDK for Interchange hubs — includes resident distiller", + "keywords": [ + "hono", + "interchange", + "knowledge", + "pgvector", + "rag", + "search" + ], + "homepage": "https://github.com/corbitsdev/corbits-memory#readme", + "bugs": { + "url": "https://github.com/corbitsdev/corbits-memory/issues" + }, + "license": "LGPL-2.1-only", + "author": "Sawyer Cutler ", + "repository": { + "type": "git", + "url": "git+https://github.com/corbitsdev/corbits-memory.git" + }, + "files": [ + "dist", + "migrations", + "README.md", + "LICENSE" + ], + "type": "module", + "sideEffects": false, "main": "./dist/index.js", "types": "./dist/index.d.ts", "exports": { @@ -26,36 +52,9 @@ "default": "./dist/distiller/index.js" } }, - "interchange": { - "grantRequirements": [ - { - "resource": "memory", - "action": "add", - "installHint": "tenant", - "surfaces": ["tools", "distiller", "routes"] - }, - { - "resource": "memory", - "action": "search", - "installHint": "tenant", - "surfaces": ["tools", "distiller", "routes"] - }, - { - "resource": "memory", - "action": "forget", - "installHint": "creator", - "surfaces": ["routes"] - }, - { - "resource": "memory", - "action": "purge", - "installHint": "creator", - "surfaces": ["routes"] - } - ] + "publishConfig": { + "access": "public" }, - "license": "LGPL-2.1-only", - "type": "module", "scripts": { "db:setup": "bun run scripts/db-setup.ts", "typecheck": "tsc --noEmit", @@ -63,24 +62,14 @@ "prepack": "bun run build", "test": "bun test ./src", "test:e2e": "bun test ./e2e", - "test:coverage": "bun test --coverage --coverage-reporter=lcov --coverage-reporter=text ./src ./e2e" + "test:coverage": "bun test --coverage --coverage-reporter=lcov --coverage-reporter=text ./src ./e2e", + "lint": "oxlint", + "format": "oxfmt", + "format:check": "oxfmt --check" }, "dependencies": { "arktype": "^2.1.29" }, - "peerDependencies": { - "@intx/agent": "^0.4.0", - "@intx/authz": "^0.4.0", - "@intx/db": "^0.4.0", - "@intx/hub-api": "^0.4.0", - "@intx/log": "^0.4.0", - "@intx/types": "^0.4.0", - "@intx/workflow": "^0.4.0", - "drizzle-orm": "^0.45.2", - "hono": "^4.12.31", - "hono-openapi": "^1.3.1", - "postgres": "^3.4.9" - }, "devDependencies": { "@corbits/memory-0.1.0": "npm:@corbits/memory@0.1.0", "@intx/agent": "0.4.0", @@ -94,34 +83,62 @@ "drizzle-orm": "0.45.2", "hono": "4.12.31", "hono-openapi": "1.3.1", + "oxfmt": "0.70.0", + "oxlint": "1.85.0", "postgres": "3.4.9", "typescript": "^5.9.0" }, - "author": "Sawyer Cutler ", - "repository": { - "type": "git", - "url": "git+https://github.com/corbitsdev/corbits-memory.git" - }, - "homepage": "https://github.com/corbitsdev/corbits-memory#readme", - "bugs": { - "url": "https://github.com/corbitsdev/corbits-memory/issues" + "peerDependencies": { + "@intx/agent": "^0.4.0", + "@intx/authz": "^0.4.0", + "@intx/db": "^0.4.0", + "@intx/hub-api": "^0.4.0", + "@intx/log": "^0.4.0", + "@intx/types": "^0.4.0", + "@intx/workflow": "^0.4.0", + "drizzle-orm": "^0.45.2", + "hono": "^4.12.31", + "hono-openapi": "^1.3.1", + "postgres": "^3.4.9" }, - "keywords": [ - "knowledge", - "rag", - "search", - "pgvector", - "interchange", - "hono" - ], - "files": [ - "dist", - "migrations", - "README.md", - "LICENSE" - ], - "sideEffects": false, - "publishConfig": { - "access": "public" + "interchange": { + "grantRequirements": [ + { + "resource": "memory", + "action": "add", + "installHint": "tenant", + "surfaces": [ + "tools", + "distiller", + "routes" + ] + }, + { + "resource": "memory", + "action": "search", + "installHint": "tenant", + "surfaces": [ + "tools", + "distiller", + "routes" + ] + }, + { + "resource": "memory", + "action": "forget", + "installHint": "creator", + "surfaces": [ + "routes" + ] + }, + { + "resource": "memory", + "action": "purge", + "installHint": "creator", + "surfaces": [ + "routes" + ] + } + ] } } diff --git a/scripts/db-setup.ts b/scripts/db-setup.ts index f7d64e4..a91634e 100644 --- a/scripts/db-setup.ts +++ b/scripts/db-setup.ts @@ -12,7 +12,10 @@ await runMemoryMigrations( user: decodeURIComponent(url.username), password: decodeURIComponent(url.password), database: url.pathname.slice(1), - ssl: sslmode === "require" || sslmode === "verify-ca" || sslmode === "verify-full", + ssl: + sslmode === "require" || + sslmode === "verify-ca" || + sslmode === "verify-full", }, { schema: "public", ftsLanguage: memory.ftsLanguage }, ); diff --git a/src/core/authority.test.ts b/src/core/authority.test.ts index d7bf171..b565c0b 100644 --- a/src/core/authority.test.ts +++ b/src/core/authority.test.ts @@ -41,7 +41,11 @@ describe("computeAuthority", () => { signals({ createdByKind: "human", actorCount: 1, sourceClass: "native" }), ); const multiPartyThread = computeAuthority( - signals({ createdByKind: "human", actorCount: 5, sourceClass: "channel" }), + signals({ + createdByKind: "human", + actorCount: 5, + sourceClass: "channel", + }), ); expect(soloArtifact).toBeLessThan(multiPartyThread); }); diff --git a/src/core/chunk/token-recursive.ts b/src/core/chunk/token-recursive.ts index 33364dc..2faeb59 100644 --- a/src/core/chunk/token-recursive.ts +++ b/src/core/chunk/token-recursive.ts @@ -1,4 +1,8 @@ -import { DEFAULT_CHUNK_CAPS, type ChunkCaps, type TokenChunk } from "./types.js"; +import { + DEFAULT_CHUNK_CAPS, + type ChunkCaps, + type TokenChunk, +} from "./types.js"; // Separator ladder tried in order before falling back to a hard character // cut. Word/whitespace-based token counting @@ -130,7 +134,10 @@ export function chunkTokenRecursive( const overlapChars = resolved.overlapTokens * CHARS_PER_TOKEN; const leaves = splitRecursive(text, 0, maxChars, 0); - const merged = mergeRemnant(mergeLeaves(leaves, maxChars, overlapChars), minChars); + const merged = mergeRemnant( + mergeLeaves(leaves, maxChars, overlapChars), + minChars, + ); return merged.map((group, ordinal) => { const chunkText = group.map((leaf) => leaf.text).join(""); diff --git a/src/core/corroboration.test.ts b/src/core/corroboration.test.ts index 104196f..b70b387 100644 --- a/src/core/corroboration.test.ts +++ b/src/core/corroboration.test.ts @@ -5,10 +5,7 @@ import { effectiveAuthority, meetsStrongEvidenceGate, } from "./corroboration.js"; -import { - BOOST_MULTIPLIER_MAX, - BOOST_MULTIPLIER_MIN, -} from "./hybrid-search.js"; +import { BOOST_MULTIPLIER_MAX, BOOST_MULTIPLIER_MIN } from "./hybrid-search.js"; describe("corroborationFactor", () => { it("is neutral with no edges", () => { diff --git a/src/core/corroboration.ts b/src/core/corroboration.ts index a07b409..2239bb4 100644 --- a/src/core/corroboration.ts +++ b/src/core/corroboration.ts @@ -78,7 +78,9 @@ export type StrongEvidenceSignals = { * Whether evidence may report strong given relevance already cleared. * Requires authority floor AND (stated human OR supports ≥ floor). */ -export function meetsStrongEvidenceGate(signals: StrongEvidenceSignals): boolean { +export function meetsStrongEvidenceGate( + signals: StrongEvidenceSignals, +): boolean { if (signals.authority < signals.authorityFloor) return false; const floor = signals.corroborationFloor ?? CORROBORATION_STRONG_FLOOR; const statedHuman = diff --git a/src/core/degrade-metrics.test.ts b/src/core/degrade-metrics.test.ts index 17617f6..3238831 100644 --- a/src/core/degrade-metrics.test.ts +++ b/src/core/degrade-metrics.test.ts @@ -1,4 +1,4 @@ -import { afterEach, beforeEach, describe, expect, it, mock } from "bun:test"; +import { afterEach, beforeEach, describe, expect, it } from "bun:test"; import { ALL_DEGRADE_FLAGS, configureDegradeMetrics, @@ -10,7 +10,11 @@ import { import { log } from "../log.js"; function captureLogs() { - const calls: Array<{ level: "info" | "warn" | "error"; message: string; payload: unknown }> = []; + const calls: Array<{ + level: "info" | "warn" | "error"; + message: string; + payload: unknown; + }> = []; const originalInfo = log.info; const originalWarn = log.warn; const originalError = log.error; @@ -85,7 +89,8 @@ describe("degrade-metrics", () => { // — but the message string alone (no context object) must still show // the numbers a human needs. for (let i = 0; i < 90; i++) recordDegrade("tenant-a", undefined); - for (let i = 0; i < 10; i++) recordDegrade("tenant-a", ["dense_unavailable"]); + for (let i = 0; i < 10; i++) + recordDegrade("tenant-a", ["dense_unavailable"]); const info = calls.find((c) => c.level === "info"); expect(info).toBeDefined(); expect(calls.some((c) => c.level === "error")).toBe(false); @@ -101,7 +106,8 @@ describe("degrade-metrics", () => { it("escalates on a saturated window (the original CL-4600 shape: 100% for 100+ calls)", () => { const { calls, restore } = captureLogs(); try { - for (let i = 0; i < 100; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); + for (let i = 0; i < 100; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); const error = calls.find((c) => c.level === "error"); expect(error).toBeDefined(); expect(error!.message).toContain("rerank_unavailable"); @@ -114,7 +120,10 @@ describe("degrade-metrics", () => { const { calls, restore } = captureLogs(); try { for (let i = 0; i < 2000; i++) { - recordDegrade("tenant-a", i % 100 === 50 ? undefined : ["rerank_unavailable"]); + recordDegrade( + "tenant-a", + i % 100 === 50 ? undefined : ["rerank_unavailable"], + ); } const errors = calls.filter((c) => c.level === "error"); expect(errors.length).toBeGreaterThan(0); @@ -131,7 +140,10 @@ describe("degrade-metrics", () => { let escalatedAtCall = -1; for (let i = 0; i < 300; i++) { recordDegrade("tenant-a", ["rerank_unavailable"]); - if (escalatedAtCall === -1 && calls.some((c) => c.level === "error")) { + if ( + escalatedAtCall === -1 && + calls.some((c) => c.level === "error") + ) { escalatedAtCall = i; } } @@ -158,12 +170,17 @@ describe("degrade-metrics", () => { it("hysteresis prevents flapping: once escalated, a dip that doesn't cross the low watermark stays escalated without re-logging error on every call", () => { const { calls, restore } = captureLogs(); try { - for (let i = 0; i < 200; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); - const escalationsAfterSaturation = calls.filter((c) => c.level === "error").length; + for (let i = 0; i < 200; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); + const escalationsAfterSaturation = calls.filter( + (c) => c.level === "error", + ).length; // A single healthy call (rate stays well above the low watermark) // should not cause a second escalation transition. recordDegrade("tenant-a", undefined); - const escalationsAfterOneHealthyCall = calls.filter((c) => c.level === "error").length; + const escalationsAfterOneHealthyCall = calls.filter( + (c) => c.level === "error", + ).length; expect(escalationsAfterOneHealthyCall).toBe(escalationsAfterSaturation); } finally { restore(); @@ -179,13 +196,17 @@ describe("degrade-metrics", () => { // without a minimum-sample guard this can cross the watermark on a // window of 1-5 calls purely from small-sample noise. for (let i = 0; i < 50; i++) { - recordDegrade("cold-tenant", i % 5 === 0 ? ["rerank_unavailable"] : undefined); + recordDegrade( + "cold-tenant", + i % 5 === 0 ? ["rerank_unavailable"] : undefined, + ); } const errorsBeforeMinSamples = calls.filter( (c) => c.level === "error" && c.message.includes("cold-tenant") && - (c.payload as { windowSize?: number } | undefined)?.windowSize !== undefined && + (c.payload as { windowSize?: number } | undefined)?.windowSize !== + undefined && (c.payload as { windowSize: number }).windowSize < 32, ); expect(errorsBeforeMinSamples.length).toBe(0); @@ -198,9 +219,14 @@ describe("degrade-metrics", () => { const { calls, restore } = captureLogs(); try { for (let i = 0; i < 200; i++) { - recordDegrade("cold-tenant", i % 5 === 0 ? ["rerank_unavailable"] : undefined); + recordDegrade( + "cold-tenant", + i % 5 === 0 ? ["rerank_unavailable"] : undefined, + ); } - const errors = calls.filter((c) => c.level === "error" && c.message.includes("cold-tenant")); + const errors = calls.filter( + (c) => c.level === "error" && c.message.includes("cold-tenant"), + ); expect(errors.length).toBeGreaterThan(0); } finally { restore(); @@ -225,7 +251,8 @@ describe("degrade-metrics", () => { for (let t = 0; t < 49; t++) { for (let i = 0; i < 50; i++) recordDegrade(`tenant-${t}`, undefined); } - for (let i = 0; i < 200; i++) recordDegrade("tenant-down", ["rerank_unavailable"]); + for (let i = 0; i < 200; i++) + recordDegrade("tenant-down", ["rerank_unavailable"]); const downErrors = calls.filter( (c) => c.level === "error" && c.message.includes("tenant-down"), @@ -242,13 +269,18 @@ describe("degrade-metrics", () => { }); it("keeps independent windows/counts per tenant", () => { - for (let i = 0; i < 10; i++) recordDegrade("tenant-a", ["dense_unavailable"]); + for (let i = 0; i < 10; i++) + recordDegrade("tenant-a", ["dense_unavailable"]); for (let i = 0; i < 5; i++) recordDegrade("tenant-b", undefined); expect(getDegradeMetricsSnapshot("tenant-a").totalSearches).toBe(10); expect(getDegradeMetricsSnapshot("tenant-b").totalSearches).toBe(5); - expect(getDegradeMetricsSnapshot("tenant-a").degradeCounts.dense_unavailable).toBe(10); - expect(getDegradeMetricsSnapshot("tenant-b").degradeCounts.dense_unavailable).toBe(0); + expect( + getDegradeMetricsSnapshot("tenant-a").degradeCounts.dense_unavailable, + ).toBe(10); + expect( + getDegradeMetricsSnapshot("tenant-b").degradeCounts.dense_unavailable, + ).toBe(0); }); it("getAllDegradeMetricsSnapshots exposes every tracked tenant for a host to poll/forward", () => { @@ -272,7 +304,8 @@ describe("degrade-metrics", () => { }); it("an idle tenant's escalated state is preserved in its snapshot even though it never re-evaluates without new calls", () => { - for (let i = 0; i < 200; i++) recordDegrade("idle-incident", ["rerank_unavailable"]); + for (let i = 0; i < 200; i++) + recordDegrade("idle-incident", ["rerank_unavailable"]); const snapshotRightAfter = getDegradeMetricsSnapshot("idle-incident"); expect(snapshotRightAfter.escalated.rerank_unavailable).toBe(true); @@ -287,7 +320,8 @@ describe("degrade-metrics", () => { it("does not evict a currently-escalated tenant to make room for new tenants", () => { configureDegradeMetrics({ maxTrackedTenants: 3 }); - for (let i = 0; i < 200; i++) recordDegrade("escalated-tenant", ["rerank_unavailable"]); + for (let i = 0; i < 200; i++) + recordDegrade("escalated-tenant", ["rerank_unavailable"]); // Fill past capacity with fresh, healthy tenants. recordDegrade("healthy-1", undefined); @@ -306,7 +340,8 @@ describe("degrade-metrics", () => { recordDegrade("tenant-a", undefined); recordDegrade("tenant-b", undefined); const evictionLog = calls.find( - (c) => c.message.includes("evicted") || c.message.includes("evicting"), + (c) => + c.message.includes("evicted") || c.message.includes("evicting"), ); expect(evictionLog).toBeDefined(); } finally { @@ -354,14 +389,17 @@ describe("degrade-metrics", () => { describe("windowed vs. cumulative snapshot (dilution)", () => { it("the snapshot exposes a windowed rate that reflects a live incident even when the cumulative rate looks tame", () => { for (let i = 0; i < 9000; i++) recordDegrade("tenant-a", undefined); - for (let i = 0; i < 1000; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); + for (let i = 0; i < 1000; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); const snapshot = getDegradeMetricsSnapshot("tenant-a"); const cumulativeRate = snapshot.degradeCounts.rerank_unavailable / snapshot.totalSearches; expect(cumulativeRate).toBeCloseTo(0.1, 1); // The windowed rate reflects the live incident, not the lifetime dilution. - expect(snapshot.windowedDegradeRate.rerank_unavailable).toBeGreaterThan(0.9); + expect(snapshot.windowedDegradeRate.rerank_unavailable).toBeGreaterThan( + 0.9, + ); expect(snapshot.escalated.rerank_unavailable).toBe(true); }); }); @@ -391,16 +429,24 @@ describe("degrade-metrics", () => { }); it("a config the validator accepts always lets a fully-degraded tenant escalate", () => { - configureDegradeMetrics({ windowSize: 50, highWatermark: 0.3, lowWatermark: 0.15 }); - for (let i = 0; i < 200; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); - expect(getDegradeMetricsSnapshot("tenant-a").escalated.rerank_unavailable).toBe(true); + configureDegradeMetrics({ + windowSize: 50, + highWatermark: 0.3, + lowWatermark: 0.15, + }); + for (let i = 0; i < 200; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); + expect( + getDegradeMetricsSnapshot("tenant-a").escalated.rerank_unavailable, + ).toBe(true); }); }); describe("reconfiguring windowSize on live tenants", () => { it("shrinking windowSize on a live tenant never produces a rate above 1.0", () => { configureDegradeMetrics({ windowSize: 200 }); - for (let i = 0; i < 200; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); + for (let i = 0; i < 200; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); // 32 is the minimum-sample floor at the default 20% watermark — any // smaller windowSize at this watermark is itself rejected by the // invariant check, so this is the smallest legal shrink target. @@ -408,29 +454,41 @@ describe("degrade-metrics", () => { recordDegrade("tenant-a", ["rerank_unavailable"]); const snapshot = getDegradeMetricsSnapshot("tenant-a"); - expect(snapshot.windowedDegradeRate.rerank_unavailable).toBeLessThanOrEqual(1); + expect( + snapshot.windowedDegradeRate.rerank_unavailable, + ).toBeLessThanOrEqual(1); expect(snapshot.windowSize).toBeLessThanOrEqual(32); }); it("growing windowSize on a live tenant also stays well-formed and still escalates once resample fills back up", () => { configureDegradeMetrics({ windowSize: 50 }); - for (let i = 0; i < 6; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); + for (let i = 0; i < 6; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); configureDegradeMetrics({ windowSize: 300 }); - for (let i = 0; i < 100; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); + for (let i = 0; i < 100; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); const snapshot = getDegradeMetricsSnapshot("tenant-a"); - expect(snapshot.windowedDegradeRate.rerank_unavailable).toBeLessThanOrEqual(1); + expect( + snapshot.windowedDegradeRate.rerank_unavailable, + ).toBeLessThanOrEqual(1); expect(snapshot.escalated.rerank_unavailable).toBe(true); }); it("a tenant that was escalated before a windowSize change re-escalates once enough post-resize samples accrue, rather than staying stuck", () => { configureDegradeMetrics({ windowSize: 200 }); - for (let i = 0; i < 200; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); - expect(getDegradeMetricsSnapshot("tenant-a").escalated.rerank_unavailable).toBe(true); + for (let i = 0; i < 200; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); + expect( + getDegradeMetricsSnapshot("tenant-a").escalated.rerank_unavailable, + ).toBe(true); configureDegradeMetrics({ windowSize: 40 }); - for (let i = 0; i < 40; i++) recordDegrade("tenant-a", ["rerank_unavailable"]); - expect(getDegradeMetricsSnapshot("tenant-a").escalated.rerank_unavailable).toBe(true); + for (let i = 0; i < 40; i++) + recordDegrade("tenant-a", ["rerank_unavailable"]); + expect( + getDegradeMetricsSnapshot("tenant-a").escalated.rerank_unavailable, + ).toBe(true); }); }); }); diff --git a/src/core/degrade-metrics.ts b/src/core/degrade-metrics.ts index 36cbf9d..18c6477 100644 --- a/src/core/degrade-metrics.ts +++ b/src/core/degrade-metrics.ts @@ -28,7 +28,6 @@ const DEGRADE_FLAG_SET = { lexical_only: true, } satisfies Record; - // Deriving the list from a `satisfies Record` object // means adding a flag in hybrid-search.ts without adding it here is a // compile error (missing property), not a silent gap that only a test @@ -214,7 +213,7 @@ function newTenantState(): TenantState { totalSearches: 0, degradeCounts: emptyDegradeCounts(), since: new Date(), - windowBuffer: new Array(config.windowSize).fill(undefined), + windowBuffer: Array.from({ length: config.windowSize }, () => undefined), windowCursor: 0, windowFilled: 0, windowFlagCounts: zeroFlagCounts(), @@ -239,7 +238,7 @@ let tenants = new Map(); // trade-off. function resizeAllTenantWindows(windowSize: number): void { for (const state of tenants.values()) { - state.windowBuffer = new Array(windowSize).fill(undefined); + state.windowBuffer = Array.from({ length: windowSize }, () => undefined); state.windowCursor = 0; state.windowFilled = 0; state.windowFlagCounts = zeroFlagCounts(); @@ -296,7 +295,10 @@ function pushToWindow(state: TenantState, flags: readonly DegradeFlag[]): void { const evicted = state.windowBuffer[state.windowCursor]; if (evicted) { for (const flag of evicted) { - state.windowFlagCounts[flag] = Math.max(0, state.windowFlagCounts[flag] - 1); + state.windowFlagCounts[flag] = Math.max( + 0, + state.windowFlagCounts[flag] - 1, + ); } } else { state.windowFilled = Math.min(state.windowFilled + 1, config.windowSize); @@ -332,7 +334,10 @@ export interface DegradeMetricsSnapshot { escalated: Record; } -function toSnapshot(tenantId: string, state: TenantState): DegradeMetricsSnapshot { +function toSnapshot( + tenantId: string, + state: TenantState, +): DegradeMetricsSnapshot { const windowedDegradeRate = {} as Record; for (const flag of ALL_DEGRADE_FLAGS) { windowedDegradeRate[flag] = windowedRate(state, flag); @@ -422,7 +427,9 @@ export function recordDegrade( } /** Read-only snapshot for a host to expose on its own metrics/health endpoint. */ -export function getDegradeMetricsSnapshot(tenantId: string): DegradeMetricsSnapshot { +export function getDegradeMetricsSnapshot( + tenantId: string, +): DegradeMetricsSnapshot { const state = tenants.get(tenantId) ?? newTenantState(); return toSnapshot(tenantId, state); } diff --git a/src/core/embed-client.ts b/src/core/embed-client.ts index 21692dd..1aa9149 100644 --- a/src/core/embed-client.ts +++ b/src/core/embed-client.ts @@ -41,7 +41,9 @@ const DEFAULT_BATCH_SIZE = 32; const PROBE_TEXT = "embedding dimension probe"; function buildHeaders(config: EmbedClientConfig): Record { - const headers: Record = { "content-type": "application/json" }; + const headers: Record = { + "content-type": "application/json", + }; if (config.apiKey) { headers.authorization = `Bearer ${config.apiKey}`; } @@ -73,7 +75,10 @@ async function doFetch( signal: AbortSignal.timeout(timeoutMs), }); } catch (err) { - if (err instanceof Error && (err.name === "TimeoutError" || err.name === "AbortError")) { + if ( + err instanceof Error && + (err.name === "TimeoutError" || err.name === "AbortError") + ) { throw new EmbedTimeoutError( `Embed request to ${url} timed out after ${timeoutMs}ms`, ); @@ -108,7 +113,11 @@ async function embedBatchOpenAi( await assertOk(res, url); const json = (await res.json()) as { data?: Array<{ embedding: number[] }> }; if (!json.data) { - throw new EmbedHttpError(res.status, "openai-compat response missing data[]", url); + throw new EmbedHttpError( + res.status, + "openai-compat response missing data[]", + url, + ); } return json.data.map((d) => d.embedding); } @@ -174,7 +183,8 @@ async function embedBatch( config: EmbedClientConfig, fetchImpl: typeof fetch, ): Promise { - if (config.apiStyle === "openai") return embedBatchOpenAi(batch, config, fetchImpl); + if (config.apiStyle === "openai") + return embedBatchOpenAi(batch, config, fetchImpl); if (config.apiStyle === "tei") return embedBatchTei(batch, config, fetchImpl); return embedBatchOllama(batch, config, fetchImpl); } diff --git a/src/core/embed-model-registry.test.ts b/src/core/embed-model-registry.test.ts index 086d45c..349a0c8 100644 --- a/src/core/embed-model-registry.test.ts +++ b/src/core/embed-model-registry.test.ts @@ -27,7 +27,11 @@ function jsonResponse(body: unknown): Response { function fixtureFetch(dims: number): typeof fetch { return mock(() => - Promise.resolve(jsonResponse({ data: [{ embedding: new Array(dims).fill(0.1) }] })), + Promise.resolve( + jsonResponse({ + data: [{ embedding: Array.from({ length: dims }, () => 0.1) }], + }), + ), ) as unknown as typeof fetch; } @@ -37,7 +41,10 @@ const baseConfig: EmbedClientConfig = { apiStyle: "openai", }; -function createMockClient(): { client: EmbedRegistrySqlClient; queries: Array<{ sql: string; params: readonly unknown[] }> } { +function createMockClient(): { + client: EmbedRegistrySqlClient; + queries: Array<{ sql: string; params: readonly unknown[] }>; +} { const queries: Array<{ sql: string; params: readonly unknown[] }> = []; const client: EmbedRegistrySqlClient = { query: (sql, params) => { @@ -83,21 +90,21 @@ describe("discoverModelDims", () => { }); it("refuses a too-small probe result (T6)", async () => { - await expect(discoverModelDims(baseConfig, fixtureFetch(32))).rejects.toThrow( - DimsOutOfBoundsError, - ); + await expect( + discoverModelDims(baseConfig, fixtureFetch(32)), + ).rejects.toThrow(DimsOutOfBoundsError); }); it("refuses a too-large probe result (T6)", async () => { - await expect(discoverModelDims(baseConfig, fixtureFetch(5000))).rejects.toThrow( - DimsOutOfBoundsError, - ); + await expect( + discoverModelDims(baseConfig, fixtureFetch(5000)), + ).rejects.toThrow(DimsOutOfBoundsError); }); it("accepts the halfvec cap exactly and rejects one past it", async () => { - expect(await discoverModelDims(baseConfig, fixtureFetch(HALFVEC_INDEX_MAX_DIMS))).toBe( - HALFVEC_INDEX_MAX_DIMS, - ); + expect( + await discoverModelDims(baseConfig, fixtureFetch(HALFVEC_INDEX_MAX_DIMS)), + ).toBe(HALFVEC_INDEX_MAX_DIMS); await expect( discoverModelDims(baseConfig, fixtureFetch(HALFVEC_INDEX_MAX_DIMS + 1)), ).rejects.toThrow(DimsOutOfBoundsError); @@ -112,7 +119,9 @@ describe("cosineDistanceExpr", () => { }); it("emits the halfvec expression past the vector index cap", () => { - expect(cosineDistanceExpr("e.embedding", "$3", VECTOR_INDEX_MAX_DIMS + 1)).toBe( + expect( + cosineDistanceExpr("e.embedding", "$3", VECTOR_INDEX_MAX_DIMS + 1), + ).toBe( `(e.embedding::halfvec(${VECTOR_INDEX_MAX_DIMS + 1})) <=> $3::halfvec(${VECTOR_INDEX_MAX_DIMS + 1})`, ); }); @@ -144,7 +153,9 @@ describe("activateEmbedModel", () => { expect(insertQuery?.params).toContain("tenant-1"); expect(insertQuery?.params).toContain(768); - const createTableQuery = queries.find((q) => q.sql.includes("CREATE TABLE IF NOT EXISTS")); + const createTableQuery = queries.find((q) => + q.sql.includes("CREATE TABLE IF NOT EXISTS"), + ); expect(createTableQuery?.sql).toContain(result.tableName); expect(createTableQuery?.sql).toContain("vector(768)"); const bare = result.tableName.replace(/^"memory"\."|"$/g, ""); @@ -153,7 +164,9 @@ describe("activateEmbedModel", () => { 'FOREIGN KEY (chunk_id) REFERENCES "memory"."chunk" (id) ON DELETE CASCADE', ); - const tenantIndexQuery = queries.find((q) => q.sql.includes("_tenant_chunk_idx")); + const tenantIndexQuery = queries.find((q) => + q.sql.includes("_tenant_chunk_idx"), + ); expect(tenantIndexQuery?.sql).toBe( `CREATE INDEX IF NOT EXISTS ${bare}_tenant_chunk_idx ON ${result.tableName} (tenant_id, chunk_id)`, ); @@ -167,7 +180,9 @@ describe("activateEmbedModel", () => { await activateEmbedModel(client, "tenant-1", baseConfig, fixtureFetch(768)); await activateEmbedModel(client, "tenant-1", baseConfig, fixtureFetch(768)); - const tenantIndexQueries = queries.filter((q) => q.sql.includes("_tenant_chunk_idx")); + const tenantIndexQueries = queries.filter((q) => + q.sql.includes("_tenant_chunk_idx"), + ); expect(tenantIndexQueries).toHaveLength(2); }); @@ -191,13 +206,25 @@ describe("activateEmbedModel", () => { it("is idempotent — calling twice with identical config computes the same table name (T5)", async () => { const { client, queries } = createMockClient(); - const first = await activateEmbedModel(client, "tenant-1", baseConfig, fixtureFetch(768)); - const second = await activateEmbedModel(client, "tenant-1", baseConfig, fixtureFetch(768)); + const first = await activateEmbedModel( + client, + "tenant-1", + baseConfig, + fixtureFetch(768), + ); + const second = await activateEmbedModel( + client, + "tenant-1", + baseConfig, + fixtureFetch(768), + ); expect(first.tableName).toBe(second.tableName); expect(first.modelKey).toBe(second.modelKey); - const createTableQueries = queries.filter((q) => q.sql.includes("CREATE TABLE IF NOT EXISTS")); + const createTableQueries = queries.filter((q) => + q.sql.includes("CREATE TABLE IF NOT EXISTS"), + ); expect(createTableQueries).toHaveLength(2); expect(createTableQueries[0]?.sql).toBe(createTableQueries[1]?.sql); }); @@ -205,7 +232,12 @@ describe("activateEmbedModel", () => { it("creates a halfvec expression hnsw index past the vector cap, matching the query expression", async () => { const dims = 3072; const { client, queries } = createMockClient(); - const result = await activateEmbedModel(client, "tenant-1", baseConfig, fixtureFetch(dims)); + const result = await activateEmbedModel( + client, + "tenant-1", + baseConfig, + fixtureFetch(dims), + ); expect(result.dims).toBe(dims); const indexQuery = queries.find((q) => q.sql.includes("USING hnsw")); @@ -269,7 +301,9 @@ describe("resolveActiveEmbedTable", () => { const modelKey = computeModelKey(baseConfig.baseUrl, baseConfig.modelId); const client: EmbedRegistrySqlClient = { query: () => - Promise.resolve([{ model_key: modelKey, model_id: baseConfig.modelId, dims: 768 }]), + Promise.resolve([ + { model_key: modelKey, model_id: baseConfig.modelId, dims: 768 }, + ]), }; const result = await resolveActiveEmbedTable(client, "tenant-1"); expect(result).toEqual({ @@ -293,15 +327,22 @@ describe("ensureEmbedModel", () => { expect(insertQuery?.sql).not.toContain("'active'"); // No UPDATE ... SET status='active' issued by ensure. - expect( - queries.some((q) => q.sql.includes("SET status = 'active'")), - ).toBe(false); + expect(queries.some((q) => q.sql.includes("SET status = 'active'"))).toBe( + false, + ); }); it("still creates the per-model table + indexes (table usable on replay)", async () => { const { client, queries } = createMockClient(); - const result = await ensureEmbedModel(client, "tenant-1", baseConfig, fixtureFetch(768)); - const createTableQuery = queries.find((q) => q.sql.includes("CREATE TABLE IF NOT EXISTS")); + const result = await ensureEmbedModel( + client, + "tenant-1", + baseConfig, + fixtureFetch(768), + ); + const createTableQuery = queries.find((q) => + q.sql.includes("CREATE TABLE IF NOT EXISTS"), + ); expect(createTableQuery?.sql).toContain(result.tableName); expect(queries.some((q) => q.sql.includes("USING hnsw"))).toBe(true); }); @@ -326,10 +367,14 @@ describe("activateEmbedModel (split)", () => { await activateEmbedModel(client, "tenant-1", baseConfig, fixtureFetch(768)); // activate always issues the exclusive active UPDATE after ensure work expect( - queries.slice(ensureQueries).some((q) => q.sql.includes("SET status = 'active'")), + queries + .slice(ensureQueries) + .some((q) => q.sql.includes("SET status = 'active'")), ).toBe(true); expect( - queries.slice(ensureQueries).some((q) => q.sql.includes("SET status = 'ready'")), + queries + .slice(ensureQueries) + .some((q) => q.sql.includes("SET status = 'ready'")), ).toBe(true); }); }); @@ -361,8 +406,12 @@ describe("activateEmbedModelByKey", () => { expect(result.modelKey).toBe(modelKey); expect(result.dims).toBe(768); expect(queries[0]?.sql).toContain("SELECT model_key"); - expect(queries.some((q) => q.sql.includes("SET status = 'ready'"))).toBe(true); - expect(queries.some((q) => q.sql.includes("SET status = 'active'"))).toBe(true); + expect(queries.some((q) => q.sql.includes("SET status = 'ready'"))).toBe( + true, + ); + expect(queries.some((q) => q.sql.includes("SET status = 'active'"))).toBe( + true, + ); }); it("throws when the registry row is missing", async () => { @@ -385,7 +434,9 @@ describe("activateEmbedModelByKey", () => { describe("resolveEmbedTableByModelKey", () => { it("returns null when no row for this model_key", async () => { const client: EmbedRegistrySqlClient = { query: () => Promise.resolve([]) }; - expect(await resolveEmbedTableByModelKey(client, "tenant-1", "abcdef0123456789")).toBeNull(); + expect( + await resolveEmbedTableByModelKey(client, "tenant-1", "abcdef0123456789"), + ).toBeNull(); }); it("returns the table info regardless of status (ready or active)", async () => { @@ -394,13 +445,19 @@ describe("resolveEmbedTableByModelKey", () => { query: () => Promise.resolve([{ model_key: modelKey, model_id: "m", dims: 512 }]), }; - const result = await resolveEmbedTableByModelKey(client, "tenant-1", modelKey); + const result = await resolveEmbedTableByModelKey( + client, + "tenant-1", + modelKey, + ); expect(result?.dims).toBe(512); expect(result?.tableName).toBe(`"memory"."embedding_${modelKey}"`); }); it("rejects an invalid model_key format", async () => { const client: EmbedRegistrySqlClient = { query: () => Promise.resolve([]) }; - expect(() => resolveEmbedTableByModelKey(client, "tenant-1", "not-a-key")).toThrow(); + expect(() => + resolveEmbedTableByModelKey(client, "tenant-1", "not-a-key"), + ).toThrow(); }); }); diff --git a/src/core/embed-model-registry.ts b/src/core/embed-model-registry.ts index d5e3637..8c60687 100644 --- a/src/core/embed-model-registry.ts +++ b/src/core/embed-model-registry.ts @@ -22,7 +22,9 @@ export function cosineDistanceExpr( dims: number, ): string { if (!Number.isInteger(dims) || dims <= 0) { - throw new Error(`cosineDistanceExpr: dims must be a positive integer, got ${dims}`); + throw new Error( + `cosineDistanceExpr: dims must be a positive integer, got ${dims}`, + ); } if (dims <= VECTOR_INDEX_MAX_DIMS) { return `${column} <=> ${vectorParam}::vector`; @@ -75,8 +77,7 @@ export const EMBED_TABLE_BARE_PATTERN = /^embedding_[a-f0-9]{16}$/; * Fully schema-qualified embedding table name for raw SQL interpolation. * Tables live under the memory schema: "memory"."embedding_". */ -export const EMBED_TABLE_NAME_PATTERN = - /^"memory"\."embedding_[a-f0-9]{16}"$/; +export const EMBED_TABLE_NAME_PATTERN = /^"memory"\."embedding_[a-f0-9]{16}"$/; // This is the only place in this module that ever interpolates a computed // identifier into raw SQL (see activateEmbedModel below) — a future change @@ -101,7 +102,10 @@ export function embeddingTableName(modelKey: string): string { // against this shape. The engine's `RawSql` (postgres-js) handle's // `sql.unsafe(query, params)` method satisfies this interface structurally. export interface EmbedRegistrySqlClient { - query: (sql: string, params: readonly unknown[]) => Promise>>; + query: ( + sql: string, + params: readonly unknown[], + ) => Promise>>; } export interface ActivateEmbedModelResult { @@ -339,7 +343,9 @@ export async function resolveEmbedTableByModelKey( modelKey: string, ): Promise { if (!/^[a-f0-9]{16}$/.test(modelKey)) { - throw new Error(`resolveEmbedTableByModelKey: invalid modelKey "${modelKey}"`); + throw new Error( + `resolveEmbedTableByModelKey: invalid modelKey "${modelKey}"`, + ); } const rows = await client.query( `SELECT model_key, model_id, dims FROM "memory"."embed_model" diff --git a/src/core/engine-client-config.ts b/src/core/engine-client-config.ts index fa0602c..5a36c1c 100644 --- a/src/core/engine-client-config.ts +++ b/src/core/engine-client-config.ts @@ -58,8 +58,6 @@ export function toRerankClientConfig( ...(rerank.maxDocChars !== undefined ? { maxDocChars: rerank.maxDocChars } : {}), - ...(rerank.timeoutMs !== undefined - ? { timeoutMs: rerank.timeoutMs } - : {}), + ...(rerank.timeoutMs !== undefined ? { timeoutMs: rerank.timeoutMs } : {}), }; } diff --git a/src/core/enums.ts b/src/core/enums.ts index dc2399c..e41d774 100644 --- a/src/core/enums.ts +++ b/src/core/enums.ts @@ -30,10 +30,7 @@ export const EDGE_REF_TYPES_DB = [ export type EdgeRefTypeDb = (typeof EDGE_REF_TYPES_DB)[number]; /** Adapter-facing edge endpoint kinds, including planning-time `native`. */ -export const EDGE_REF_TYPES_ADAPTER = [ - ...EDGE_REF_TYPES_DB, - "native", -] as const; +export const EDGE_REF_TYPES_ADAPTER = [...EDGE_REF_TYPES_DB, "native"] as const; export type EdgeRefTypeAdapter = (typeof EDGE_REF_TYPES_ADAPTER)[number]; /** @@ -75,8 +72,6 @@ export const RETENTION_CLASSES = [ export type RetentionClass = (typeof RETENTION_CLASSES)[number]; /** Build an arktype union string from a const string array. */ -export function arktypeStringUnion( - values: readonly string[], -): string { +export function arktypeStringUnion(values: readonly string[]): string { return values.map((v) => `'${v}'`).join("|"); } diff --git a/src/core/fts-language.test.ts b/src/core/fts-language.test.ts index 9c53b33..bfb4ced 100644 --- a/src/core/fts-language.test.ts +++ b/src/core/fts-language.test.ts @@ -32,7 +32,13 @@ describe("parseFtsLanguage", () => { describe("the baseline migration language token", () => { it("is present in the generated-column DDL, ready for substitution", async () => { const ddl = await readFile( - join(import.meta.dir, "..", "..", "migrations", "0002_memory_baseline.sql"), + join( + import.meta.dir, + "..", + "..", + "migrations", + "0002_memory_baseline.sql", + ), "utf8", ); expect(ddl).toContain(`to_tsvector('${FTS_LANGUAGE_TOKEN}', "text")`); @@ -74,7 +80,9 @@ describe("verifyFtsLanguage", () => { return { query: (sqlText: string) => { if (sqlText.includes("pg_ts_config")) { - return Promise.resolve(opts.known === false ? [] : [{ "?column?": 1 }]); + return Promise.resolve( + opts.known === false ? [] : [{ "?column?": 1 }], + ); } return Promise.resolve(opts.expr == null ? [] : [{ expr: opts.expr }]); }, @@ -89,7 +97,10 @@ describe("verifyFtsLanguage", () => { it("throws when the configured language is not an installed config", async () => { await expect( - verifyFtsLanguage(fakeClient({ known: false, expr: ENGLISH_EXPR }), "klingon"), + verifyFtsLanguage( + fakeClient({ known: false, expr: ENGLISH_EXPR }), + "klingon", + ), ).rejects.toThrow("not an installed"); }); @@ -123,7 +134,9 @@ describe("createFtsVerification", () => { query: (sqlText: string) => { calls += 1; return Promise.resolve( - sqlText.includes("pg_ts_config") ? [{ ok: 1 }] : [{ expr: ENGLISH_EXPR }], + sqlText.includes("pg_ts_config") + ? [{ ok: 1 }] + : [{ expr: ENGLISH_EXPR }], ); }, }, @@ -145,7 +158,9 @@ describe("createFtsVerification", () => { return Promise.reject(new Error("connection refused")); } return Promise.resolve( - sqlText.includes("pg_ts_config") ? [{ ok: 1 }] : [{ expr: ENGLISH_EXPR }], + sqlText.includes("pg_ts_config") + ? [{ ok: 1 }] + : [{ expr: ENGLISH_EXPR }], ); }, }, @@ -160,7 +175,9 @@ describe("createFtsVerification", () => { { query: (sqlText: string) => Promise.resolve( - sqlText.includes("pg_ts_config") ? [{ ok: 1 }] : [{ expr: ENGLISH_EXPR }], + sqlText.includes("pg_ts_config") + ? [{ ok: 1 }] + : [{ expr: ENGLISH_EXPR }], ), }, "german", @@ -182,7 +199,9 @@ describe("createFtsVerification concurrency", () => { query: async (sqlText: string) => { calls += 1; await gate; - return sqlText.includes("pg_ts_config") ? [{ ok: 1 }] : [{ expr: ENGLISH_EXPR }]; + return sqlText.includes("pg_ts_config") + ? [{ ok: 1 }] + : [{ expr: ENGLISH_EXPR }]; }, }, "english", @@ -204,7 +223,9 @@ describe("createFtsVerification concurrency", () => { return Promise.reject(new Error("connection refused")); } return Promise.resolve( - sqlText.includes("pg_ts_config") ? [{ ok: 1 }] : [{ expr: ENGLISH_EXPR }], + sqlText.includes("pg_ts_config") + ? [{ ok: 1 }] + : [{ expr: ENGLISH_EXPR }], ); }, }, diff --git a/src/core/fts-language.ts b/src/core/fts-language.ts index 35556bd..b39f811 100644 --- a/src/core/fts-language.ts +++ b/src/core/fts-language.ts @@ -92,7 +92,10 @@ function rebuildColumnRecipe(language: string): string { } export interface FtsVerifySqlClient { - query: (sql: string, params: readonly unknown[]) => Promise>>; + query: ( + sql: string, + params: readonly unknown[], + ) => Promise>>; } /** @@ -127,7 +130,7 @@ export async function verifyFtsLanguage( const expr = rows[0]?.["expr"]; if (typeof expr !== "string") { throw new Error( - 'memory.chunk.text_fts has no generation expression — schema not migrated?', + "memory.chunk.text_fts has no generation expression — schema not migrated?", ); } // Only unqualified `pg_catalog` configs are supported: FTS_LANGUAGE_PATTERN diff --git a/src/core/hash.test.ts b/src/core/hash.test.ts index f06dca1..c1a5e87 100644 --- a/src/core/hash.test.ts +++ b/src/core/hash.test.ts @@ -33,7 +33,10 @@ describe("contentHash", () => { it("changes when chunk text changes", () => { const h1 = contentHash(base); - const h2 = contentHash({ ...base, chunkTexts: ["first chunk", "different"] }); + const h2 = contentHash({ + ...base, + chunkTexts: ["first chunk", "different"], + }); expect(h1).not.toBe(h2); }); diff --git a/src/core/hybrid-search.test.ts b/src/core/hybrid-search.test.ts index 176da90..52d933f 100644 --- a/src/core/hybrid-search.test.ts +++ b/src/core/hybrid-search.test.ts @@ -210,7 +210,9 @@ describe("temporalRecencyMultiplier", () => { const old = new Date("2020-01-01T00:00:00.000Z"); const far = new Date(now.getTime() + DEADLINE_LOOKAHEAD_MS * 2); type Args = Parameters[0]; - const cases: Array> = [ + const cases: Array< + Pick + > = [ { temporalClass: "state", occurredAt: old, validUntil: null }, { temporalClass: "lesson", occurredAt: old, validUntil: null }, { temporalClass: "deadline", occurredAt: now, validUntil: far }, diff --git a/src/core/merge-local-live.test.ts b/src/core/merge-local-live.test.ts index 7c61e03..d34930c 100644 --- a/src/core/merge-local-live.test.ts +++ b/src/core/merge-local-live.test.ts @@ -214,9 +214,7 @@ describe("mergeLocalLiveV1", () => { local({ externalRef: "1", score: 3 }), local({ externalRef: "2", score: 2 }), ], - live: [ - live({ externalRef: "3", score: 1, title: "third" }), - ], + live: [live({ externalRef: "3", score: 1, title: "third" })], limit: 2, nowMs: NOW, }); diff --git a/src/core/merge-local-live.ts b/src/core/merge-local-live.ts index a718bfe..1572669 100644 --- a/src/core/merge-local-live.ts +++ b/src/core/merge-local-live.ts @@ -14,9 +14,7 @@ import type { SearchHitCitation } from "./schemas/search.js"; export const LIVE_TIMEOUT_MS = 800; -export type MergeDegradeFlag = - | "live_timeout" - | "live_error"; +export type MergeDegradeFlag = "live_timeout" | "live_error"; export type MergeChannelItem = { channel: "local" | "live"; @@ -185,9 +183,11 @@ export function withTimeout( ): Promise { return new Promise((resolve, reject) => { const timer = setTimeout(() => { - reject(Object.assign(new Error(`${label} timed out after ${ms}ms`), { - code: "live_timeout" as const, - })); + reject( + Object.assign(new Error(`${label} timed out after ${ms}ms`), { + code: "live_timeout" as const, + }), + ); }, ms); promise.then( (v) => { diff --git a/src/core/rerank-client.ts b/src/core/rerank-client.ts index 8a54526..66250c5 100644 --- a/src/core/rerank-client.ts +++ b/src/core/rerank-client.ts @@ -169,7 +169,8 @@ function resolveRerankModel(config: Pick): string { function tokenLimitForModel(model: string): number { return ( - KNOWN_TEI_RERANK_MODEL_TOKEN_LIMITS[model] ?? CONSERVATIVE_FALLBACK_TOKEN_LIMIT + KNOWN_TEI_RERANK_MODEL_TOKEN_LIMITS[model] ?? + CONSERVATIVE_FALLBACK_TOKEN_LIMIT ); } @@ -191,9 +192,8 @@ export function defaultMaxDocCharsForModel(model: string): number { // The default budget for the engine's own default TEI model // (`DEFAULT_RERANK_MODEL`, bge-reranker-v2-m3) — exported for callers and // tests that want "the real default" without re-deriving it. -export const DEFAULT_MAX_DOC_CHARS = defaultMaxDocCharsForModel( - DEFAULT_RERANK_MODEL, -); +export const DEFAULT_MAX_DOC_CHARS = + defaultMaxDocCharsForModel(DEFAULT_RERANK_MODEL); export class RerankConfigError extends Error { constructor(message: string) { @@ -257,7 +257,10 @@ function truncateForRerank(text: string, budget: number): string { // resolves (see `RerankQueryTooLongError`). Returns null when the query // alone leaves less than MIN_DOC_CHARS for the document; callers must treat // null as "skip reranking for this request", not "use MIN_DOC_CHARS anyway". -function resolveDocBudget(maxDocChars: number, queryChars: number): number | null { +function resolveDocBudget( + maxDocChars: number, + queryChars: number, +): number | null { const budget = maxDocChars - queryChars; return budget < MIN_DOC_CHARS ? null : budget; } diff --git a/src/core/schemas/adapted-document.ts b/src/core/schemas/adapted-document.ts index 8ff1ab6..acb3632 100644 --- a/src/core/schemas/adapted-document.ts +++ b/src/core/schemas/adapted-document.ts @@ -70,7 +70,9 @@ export const AdaptedDocumentSchema = type({ "attributes?": "Record", entityHints: EntityHintSchema.array(), "edges?": MemoryEdgeHintSchema.array(), - chunks: AdaptedDocumentChunkSchema.array().atMostLength(MAX_CHUNKS_PER_DOCUMENT), + chunks: AdaptedDocumentChunkSchema.array().atMostLength( + MAX_CHUNKS_PER_DOCUMENT, + ), "rawPointer?": RawPointerSchema, "actor?": ActorAttributionSchema, // The raw authority signals a caller can observe. All three are optional: a diff --git a/src/core/schemas/document.ts b/src/core/schemas/document.ts index 5b5c856..c498312 100644 --- a/src/core/schemas/document.ts +++ b/src/core/schemas/document.ts @@ -26,14 +26,14 @@ export const ProvenanceModeSchema = type( export type ProvenanceMode = typeof ProvenanceModeSchema.infer; export const TemporalClassSchema = type( - arktypeStringUnion(TEMPORAL_CLASSES) as - "'event'|'deadline'|'state'|'lesson'", + arktypeStringUnion(TEMPORAL_CLASSES) as "'event'|'deadline'|'state'|'lesson'", ); export type TemporalClass = typeof TemporalClassSchema.infer; export const RetentionClassSchema = type( - arktypeStringUnion(RETENTION_CLASSES) as - "'durable'|'standard'|'ephemeral'|'source_only'", + arktypeStringUnion( + RETENTION_CLASSES, + ) as "'durable'|'standard'|'ephemeral'|'source_only'", ); export type RetentionClass = typeof RetentionClassSchema.infer; diff --git a/src/core/schemas/entity-edge.ts b/src/core/schemas/entity-edge.ts index ea080dd..aba817d 100644 --- a/src/core/schemas/entity-edge.ts +++ b/src/core/schemas/entity-edge.ts @@ -21,14 +21,16 @@ export type MemoryEntity = typeof MemoryEntitySchema.infer; // to an entity row at the capture write boundary; it is never stored on // memory.edge. export const MemoryEdgeRefTypeSchema = type( - arktypeStringUnion(EDGE_REF_TYPES_ADAPTER) as - "'document'|'version'|'chunk'|'entity'|'native'", + arktypeStringUnion( + EDGE_REF_TYPES_ADAPTER, + ) as "'document'|'version'|'chunk'|'entity'|'native'", ); export type MemoryEdgeRefType = typeof MemoryEdgeRefTypeSchema.infer; export const MemoryEdgeRelSchema = type( - arktypeStringUnion(EDGE_RELS) as - "'mentions'|'about'|'authored_by'|'involves'|'part_of'|'derived_from'|'supports'|'contradicts'|'supersedes'", + arktypeStringUnion( + EDGE_RELS, + ) as "'mentions'|'about'|'authored_by'|'involves'|'part_of'|'derived_from'|'supports'|'contradicts'|'supersedes'", ); export type MemoryEdgeRel = typeof MemoryEdgeRelSchema.infer; diff --git a/src/core/schemas/non-blank-id.ts b/src/core/schemas/non-blank-id.ts index 0381b88..0c4cfb7 100644 --- a/src/core/schemas/non-blank-id.ts +++ b/src/core/schemas/non-blank-id.ts @@ -11,5 +11,6 @@ import { type } from "arktype"; * fix lives in one schema instead of a comment repeated at each call site. */ export const NonBlankId = type("string").narrow( - (s, ctx) => s.trim().length > 0 || ctx.mustBe("non-blank (not just whitespace)"), + (s, ctx) => + s.trim().length > 0 || ctx.mustBe("non-blank (not just whitespace)"), ); diff --git a/src/core/schemas/transform.ts b/src/core/schemas/transform.ts index c7bf0b8..466cf93 100644 --- a/src/core/schemas/transform.ts +++ b/src/core/schemas/transform.ts @@ -74,7 +74,5 @@ export const ReplayRequestSchema = type({ }); export type ReplayRequest = typeof ReplayRequestSchema.infer; -export const TransformRunStatusSchema = type( - "'running'|'completed'|'failed'", -); +export const TransformRunStatusSchema = type("'running'|'completed'|'failed'"); export type TransformRunStatus = typeof TransformRunStatusSchema.infer; diff --git a/src/db/schema.ts b/src/db/schema.ts index cdd7198..dcb5637 100644 --- a/src/db/schema.ts +++ b/src/db/schema.ts @@ -134,9 +134,7 @@ export const memoryChunk = memorySchema.table( role: text("role"), createdAt: timestamp("created_at").notNull().defaultNow(), }, - (t) => [ - uniqueIndex("chunk_version_ordinal_uniq").on(t.versionId, t.ordinal), - ], + (t) => [uniqueIndex("chunk_version_ordinal_uniq").on(t.versionId, t.ordinal)], ); export const memoryEntity = memorySchema.table( diff --git a/src/distiller/tick.ts b/src/distiller/tick.ts index 51e54cc..11af62a 100644 --- a/src/distiller/tick.ts +++ b/src/distiller/tick.ts @@ -88,7 +88,8 @@ function parseFeedPage(raw: unknown): DistillTickPage { status: String(row["status"] ?? "active"), createdByKind: String(row["createdByKind"] ?? "system"), generatorAgentId: - row["generatorAgentId"] === null || row["generatorAgentId"] === undefined + row["generatorAgentId"] === null || + row["generatorAgentId"] === undefined ? null : String(row["generatorAgentId"]), provenance: String(row["provenance"] ?? "unknown"), @@ -113,8 +114,7 @@ function parseFeedPage(raw: unknown): DistillTickPage { export async function runDistillTick( args: RunDistillTickArgs, ): Promise { - const generatorAgentId = - args.generatorAgentId ?? RESIDENT_DISTILLER_AGENT_ID; + const generatorAgentId = args.generatorAgentId ?? RESIDENT_DISTILLER_AGENT_ID; const after = args.after ?? 0; const raw = await args.client.feed( diff --git a/src/grant-requirements.ts b/src/grant-requirements.ts index 5f8f75e..bff05ab 100644 --- a/src/grant-requirements.ts +++ b/src/grant-requirements.ts @@ -67,9 +67,7 @@ export const MEMORY_CAPABILITY_IDS = MEMORY_GRANT_REQUIREMENTS.map( * must not inherit a routes-only capability (like `forget`/`purge`) just * because it appears somewhere in the full requirement list. */ -export function capabilityIdsForSurface( - surface: MemoryGrantSurface, -): string[] { +export function capabilityIdsForSurface(surface: MemoryGrantSurface): string[] { return MEMORY_GRANT_REQUIREMENTS.filter((r) => r.surfaces.includes(surface), ).map((r) => `${r.resource}:${r.action}`); diff --git a/src/grant-tags.test.ts b/src/grant-tags.test.ts index b9f12dc..e58e555 100644 --- a/src/grant-tags.test.ts +++ b/src/grant-tags.test.ts @@ -9,7 +9,6 @@ import { tenantTag, } from "./grant-tags.js"; - describe("resolveAccessTags", () => { test("always includes owner tag", () => { expect(resolveAccessTags({ principalId: "u1", tenantId: "t1" })).toEqual([ diff --git a/src/http-bodies.ts b/src/http-bodies.ts index f21380b..e6c9a52 100644 --- a/src/http-bodies.ts +++ b/src/http-bodies.ts @@ -42,7 +42,9 @@ export type AddRequest = typeof AddRequest.infer; export const SearchRequest = type({ query: "string >= 1", - "limit?": type(`${SEARCH_LIMIT_MIN} <= number.integer <= ${SEARCH_LIMIT_MAX}`), + "limit?": type( + `${SEARCH_LIMIT_MIN} <= number.integer <= ${SEARCH_LIMIT_MAX}`, + ), "kinds?": "string[]", "entity_ids?": "string[]", "sources?": "string[]", @@ -75,11 +77,7 @@ export function parseListLimitString( ): number | undefined | null { if (raw === undefined || raw === "") return undefined; const n = Number(raw); - if ( - !Number.isInteger(n) || - n < LIST_LIMIT_MIN || - n > LIST_LIMIT_MAX - ) { + if (!Number.isInteger(n) || n < LIST_LIMIT_MIN || n > LIST_LIMIT_MAX) { return null; } return n; diff --git a/src/http-client.ts b/src/http-client.ts index 1e3c577..bd5bbb7 100644 --- a/src/http-client.ts +++ b/src/http-client.ts @@ -103,10 +103,9 @@ export function createMemoryHttpClient( try { return JSON.parse(text) as unknown; } catch (cause) { - throw new Error( - `memory HTTP ${res.status}: invalid JSON response`, - { cause }, - ); + throw new Error(`memory HTTP ${res.status}: invalid JSON response`, { + cause, + }); } } @@ -154,4 +153,3 @@ export function createMemoryHttpClient( }, }; } - diff --git a/src/memory.ts b/src/memory.ts index 47eecd8..477bf30 100644 --- a/src/memory.ts +++ b/src/memory.ts @@ -1,5 +1,3 @@ -import { authorize } from "@intx/authz"; - import { canAccessDocument, matchesVisibleTags, @@ -11,11 +9,17 @@ import { import type { EngineConfig } from "./config.js"; import { formatCaughtError, log } from "./log.js"; import { createDb, type Db, type RawSql } from "./db/client.js"; -import { createFtsVerification, parseFtsLanguage } from "./core/fts-language.js"; +import { + createFtsVerification, + parseFtsLanguage, +} from "./core/fts-language.js"; import { createRawSqlClient } from "./core/embed-sql.js"; import type { SearchHit } from "./core/schemas/search.js"; import { validateRerankConfig } from "./core/rerank-client.js"; -import { captureDocument, type CaptureDegradedReason } from "./services/capture.js"; +import { + captureDocument, + type CaptureDegradedReason, +} from "./services/capture.js"; import { hybridSearch, MemorySearchInputError, @@ -23,10 +27,7 @@ import { type HybridSearchResult, DEFAULT_HYBRID_TOP_K, } from "./services/search.js"; -import { - listTimelineEvents, - type TimelineEvent, -} from "./services/timeline.js"; +import { listTimelineEvents, type TimelineEvent } from "./services/timeline.js"; import { fetchFeed, feedPageAfterAccessFilter, @@ -60,10 +61,11 @@ import { MEMORY_SHARE_CONDITION_REGISTRY, } from "./services/share-grants.js"; -import { - isWritableGrantStore, -} from "./ports/writable-grant-store.js"; -import type { TransformConfigParams, TransformScope } from "./core/schemas/transform.js"; +import { isWritableGrantStore } from "./ports/writable-grant-store.js"; +import type { + TransformConfigParams, + TransformScope, +} from "./core/schemas/transform.js"; import { LIVE_TIMEOUT_MS, mergeLocalLiveV1, @@ -511,7 +513,9 @@ function resolveAddAccessTags(params: MemoryAddParams): string[] { return resolveAccessTags({ principalId: params.principalId, tenantId: params.tenantId, - ...(params.accessTags !== undefined ? { accessTags: params.accessTags } : {}), + ...(params.accessTags !== undefined + ? { accessTags: params.accessTags } + : {}), ...(params.share !== undefined ? { share: params.share } : {}), }); } @@ -859,10 +863,7 @@ function createPlaneFromStore( const hasContent = params.content !== undefined; const hasFile = params.file !== undefined; if (hasContent === hasFile) { - throw new MemoryError( - 400, - "provide exactly one of content or file", - ); + throw new MemoryError(400, "provide exactly one of content or file"); } let title: string; @@ -884,8 +885,7 @@ function createPlaneFromStore( ...(file.filename !== undefined ? { filename: file.filename } : {}), }); text = extracted.text; - title = - file.title ?? extracted.title ?? file.filename ?? "untitled"; + title = file.title ?? extracted.title ?? file.filename ?? "untitled"; } const accessTags = resolveAddAccessTags(params); @@ -947,7 +947,9 @@ function createPlaneFromStore( const docTag = documentTag(result.documentId); let tagStamped = false; if (store.appendAccessTags) { - await store.appendAccessTags(params.tenantId, result.documentId, [docTag]); + await store.appendAccessTags(params.tenantId, result.documentId, [ + docTag, + ]); tagStamped = true; } else { log.warn( @@ -1002,10 +1004,7 @@ function createPlaneFromStore( async feed(params) { if (!store.feed) { - throw new MemoryError( - 501, - "feed requires the engine DocumentStore", - ); + throw new MemoryError(501, "feed requires the engine DocumentStore"); } return store.feed({ tenantId: params.tenantId, @@ -1101,10 +1100,7 @@ function createPlaneFromStore( throw new MemoryError(404, "document not found"); } if (!isOwner(owner, input.principalId)) { - throw new MemoryError( - 403, - "only the document's creator may forget it", - ); + throw new MemoryError(403, "only the document's creator may forget it"); } return tombstoneDocument(transformDeps.db, input); }, @@ -1121,10 +1117,7 @@ function createPlaneFromStore( throw new MemoryError(404, "document not found"); } if (!isOwner(owner, input.principalId)) { - throw new MemoryError( - 403, - "only the document's creator may purge it", - ); + throw new MemoryError(403, "only the document's creator may purge it"); } return hardDeleteDocument(transformDeps.db, input); }, @@ -1252,9 +1245,7 @@ function createEngineDocumentStore(config: MemoryConfig): { }); if (result.hits.length === 0) return result; - const docIds = Array.from( - new Set(result.hits.map((h) => h.document_id)), - ); + const docIds = Array.from(new Set(result.hits.map((h) => h.document_id))); // Load access_tags + active-version creator for grant-tag check. // Raw SQL so unit tests can mock `sql` without a real drizzle client. diff --git a/src/ports/fakes.ts b/src/ports/fakes.ts index a15f026..38fb25b 100644 --- a/src/ports/fakes.ts +++ b/src/ports/fakes.ts @@ -50,7 +50,9 @@ async function visibleTo( principalId: params.principalId, createdByPrincipalId: doc.principalId, accessTags: doc.accessTags, - ...(params.visibleTags !== undefined ? { visibleTags: params.visibleTags } : {}), + ...(params.visibleTags !== undefined + ? { visibleTags: params.visibleTags } + : {}), ...(params.conditionRegistry !== undefined ? { conditionRegistry: params.conditionRegistry } : {}), diff --git a/src/ports/index.ts b/src/ports/index.ts index abf620a..54b633b 100644 --- a/src/ports/index.ts +++ b/src/ports/index.ts @@ -4,7 +4,4 @@ export type { LiveSearchItem, SourceProvider, } from "./types.js"; -export { - createFakeDocumentStore, - createFakeSourceProvider, -} from "./fakes.js"; +export { createFakeDocumentStore, createFakeSourceProvider } from "./fakes.js"; diff --git a/src/ports/merge-plane.test.ts b/src/ports/merge-plane.test.ts index 27dcbc9..d846a8e 100644 --- a/src/ports/merge-plane.test.ts +++ b/src/ports/merge-plane.test.ts @@ -10,11 +10,7 @@ import type { LiveSearchItem } from "./types.js"; const TENANT = "t_merge"; const PRINCIPAL = "p_merge"; -function liveHit( - ref: string, - title: string, - score: number, -): LiveSearchItem { +function liveHit(ref: string, title: string, score: number): LiveSearchItem { return { adapter: "linear", externalRef: ref, @@ -36,9 +32,7 @@ describe("plane merge (MergeLocalLiveV1)", () => { const plane = createMemory({ documentStore: store, sources: [ - createFakeSourceProvider("linear", [ - liveHit("CL-1", "live only", 0.9), - ]), + createFakeSourceProvider("linear", [liveHit("CL-1", "live only", 0.9)]), ], }); @@ -54,7 +48,7 @@ describe("plane merge (MergeLocalLiveV1)", () => { query: "ports", includeEvidence: true, }); - // Local fake matches "ports"; live catalog matches "ports" in title? + // Local fake matches "ports"; live catalog matches "ports" in title? // "live only" does not — add a ports live hit expect(result.items.some((i) => i.title === "local note")).toBe(true); await plane.close(); @@ -104,9 +98,9 @@ describe("plane merge (MergeLocalLiveV1)", () => { query: "ports foundation", sources: ["local"], }); - expect(result.items.every((i) => i.documentId.startsWith("fake_doc_"))).toBe( - true, - ); + expect( + result.items.every((i) => i.documentId.startsWith("fake_doc_")), + ).toBe(true); expect(result.items.some((i) => i.documentId === "CL-42")).toBe(false); await plane.close(); }); @@ -200,8 +194,7 @@ describe("plane merge (MergeLocalLiveV1)", () => { // Fake store citation.external_ref is externalRef ?? documentId. // Live also uses CL-7 with adapter fake → prefer local. const hit = result.items.find( - (i) => - i.snippet.includes("local") || i.title.includes("local"), + (i) => i.snippet.includes("local") || i.title.includes("local"), ); expect(hit).toBeDefined(); expect(hit?.snippet).toContain("local"); diff --git a/src/routes/add.ts b/src/routes/add.ts index e69f844..7622999 100644 --- a/src/routes/add.ts +++ b/src/routes/add.ts @@ -9,12 +9,7 @@ import { AddRequest } from "../http-bodies.js"; import { MemoryError } from "../memory.js"; import type { RouteDeps } from "./deps.js"; -import { - caller, - grantGuard, - requirePrincipal, - resolveCaller, -} from "./deps.js"; +import { caller, grantGuard, requirePrincipal, resolveCaller } from "./deps.js"; const AddResponse = type({ documentId: "string", @@ -103,10 +98,7 @@ export function mountAddRoute(app: Hono, deps: RouteDeps): void { }); } catch (err) { if (err instanceof MemoryError) { - return c.json( - { error: err.message }, - err.status as 400 | 501, - ); + return c.json({ error: err.message }, err.status as 400 | 501); } const errMessage = formatCaughtError(err); log.error(`memory add failed: ${errMessage}`, { error: errMessage }); diff --git a/src/routes/deps.test.ts b/src/routes/deps.test.ts index d9d181c..365a1f2 100644 --- a/src/routes/deps.test.ts +++ b/src/routes/deps.test.ts @@ -1,6 +1,4 @@ import { describe, expect, test } from "bun:test"; -import { createInMemoryGrantStore } from "@intx/authz"; -import type { GrantRule } from "@intx/authz"; import type { Context } from "hono"; import type { RequireGrant, TenantEnv } from "@intx/hub-api"; @@ -13,20 +11,6 @@ import { type RouteDeps, } from "./deps.js"; -function grant(principalId: string, action: string): GrantRule { - return { - id: `g-${action}`, - resource: "memory", - action, - effect: "allow", - origin: "role", - conditions: null, - expiresAt: null, - roleId: null, - principalId, - }; -} - const noopRequireGrant: RequireGrant = () => (async () => {}) as never; // Minimal RouteDeps for unit tests — routes here only touch requireGrant. @@ -105,7 +89,7 @@ describe("grantGuard", () => { called = { resource: String(resource), action }; return (async () => {}) as never; }; -grantGuard(deps(requireGrant), "add"); + grantGuard(deps(requireGrant), "add"); expect(called).toEqual({ resource: "memory", action: "add" }); }); }); @@ -210,7 +194,10 @@ describe("resolveCaller", () => { // guard for it at this boundary. ["whitespace-only", { tenantId: " ", principalId: "\t\n" }], // Cast past the type system the way a buggy host's JS resolver would. - ["missing-principalId", { tenantId: "tenant-run" } as unknown as ResolvedCaller], + [ + "missing-principalId", + { tenantId: "tenant-run" } as unknown as ResolvedCaller, + ], ["non-object", "tenant-run" as unknown as ResolvedCaller], ] as const)( "rejects a %s resolved caller with 500, never seating it", diff --git a/src/routes/feed.ts b/src/routes/feed.ts index 9ebaf17..7ee23bb 100644 --- a/src/routes/feed.ts +++ b/src/routes/feed.ts @@ -7,12 +7,7 @@ import { formatCaughtError, log } from "../log.js"; import { FeedQuery, parseFeedQuery } from "../http-bodies.js"; import { MemoryError } from "../memory.js"; import type { RouteDeps } from "./deps.js"; -import { - caller, - grantGuard, - requirePrincipal, - resolveCaller, -} from "./deps.js"; +import { caller, grantGuard, requirePrincipal, resolveCaller } from "./deps.js"; const FeedResponse = type({ entries: type({ @@ -75,10 +70,7 @@ export function mountFeedRoute(app: Hono, deps: RouteDeps): void { return c.json(result); } catch (err) { if (err instanceof MemoryError) { - return c.json( - { error: err.message }, - err.status as 400 | 501, - ); + return c.json({ error: err.message }, err.status as 400 | 501); } const errMessage = formatCaughtError(err); log.error(`memory feed failed: ${errMessage}`, { diff --git a/src/routes/list.ts b/src/routes/list.ts index 3188803..b49b4ad 100644 --- a/src/routes/list.ts +++ b/src/routes/list.ts @@ -5,18 +5,9 @@ import { type } from "arktype"; import { formatCaughtError, log } from "../log.js"; import { ListQuery, parseListLimitString } from "../http-bodies.js"; -import { - MemoryError, - LIST_LIMIT_MAX, - LIST_LIMIT_MIN, -} from "../memory.js"; +import { MemoryError, LIST_LIMIT_MAX, LIST_LIMIT_MIN } from "../memory.js"; import type { RouteDeps } from "./deps.js"; -import { - caller, - grantGuard, - requirePrincipal, - resolveCaller, -} from "./deps.js"; +import { caller, grantGuard, requirePrincipal, resolveCaller } from "./deps.js"; const ListResponse = type({ events: type({ diff --git a/src/routes/retention.ts b/src/routes/retention.ts index 2d7602d..6ea549b 100644 --- a/src/routes/retention.ts +++ b/src/routes/retention.ts @@ -24,18 +24,16 @@ import { } from "../http-bodies.js"; import { MemoryError } from "../memory.js"; import type { RouteDeps } from "./deps.js"; -import { - caller, - grantGuard, - requirePrincipal, - resolveCaller, -} from "./deps.js"; +import { caller, grantGuard, requirePrincipal, resolveCaller } from "./deps.js"; function respondRetentionError(err: unknown, action: string) { const errMessage = formatCaughtError(err); log.error(`memory ${action} failed: ${errMessage}`, { error: errMessage }); if (err instanceof MemoryError) { - return { body: { error: err.message }, status: err.status as 403 | 404 | 501 }; + return { + body: { error: err.message }, + status: err.status as 403 | 404 | 501, + }; } return { body: { error: `${action} failed` }, status: 502 as const }; } @@ -63,7 +61,8 @@ export function mountForgetRoute(app: Hono, deps: RouteDeps): void { describeRoute({ tags: ["memory"], - summary: "Tombstone a document — stops appearing in search, chunk text is redacted (not archived), version rows stay for audit", + summary: + "Tombstone a document — stops appearing in search, chunk text is redacted (not archived), version rows stay for audit", responses: { 200: { description: "Tombstoned", @@ -88,7 +87,10 @@ export function mountForgetRoute(app: Hono, deps: RouteDeps): void { const { reason } = c.req.valid("json"); const { scopeId, subjectId } = caller(c); if (!deps.memory.tombstoneDocument) { - return c.json({ error: "retention APIs require the engine DocumentStore" }, 501); + return c.json( + { error: "retention APIs require the engine DocumentStore" }, + 501, + ); } try { const result = await deps.memory.tombstoneDocument({ @@ -112,7 +114,8 @@ export function mountPurgeRoute(app: Hono, deps: RouteDeps): void { describeRoute({ tags: ["memory"], - summary: "Hard-delete a document — irreversible; refused while a durable version is untombstoned", + summary: + "Hard-delete a document — irreversible; refused while a durable version is untombstoned", responses: { 200: { description: "Deletion result (deleted may be false with a reason)", @@ -135,7 +138,10 @@ export function mountPurgeRoute(app: Hono, deps: RouteDeps): void { const { documentId } = c.req.valid("param"); const { scopeId, subjectId } = caller(c); if (!deps.memory.hardDeleteDocument) { - return c.json({ error: "retention APIs require the engine DocumentStore" }, 501); + return c.json( + { error: "retention APIs require the engine DocumentStore" }, + 501, + ); } try { const result = await deps.memory.hardDeleteDocument({ @@ -165,7 +171,8 @@ export function mountSetRetentionClassRoute( describeRoute({ tags: ["memory"], - summary: "Set a version's retention class (durable/standard/ephemeral/source_only)", + summary: + "Set a version's retention class (durable/standard/ephemeral/source_only)", responses: { 200: { description: "Updated", @@ -193,7 +200,10 @@ export function mountSetRetentionClassRoute( const { retention_class } = c.req.valid("json"); const { scopeId, subjectId } = caller(c); if (!deps.memory.setRetentionClass) { - return c.json({ error: "retention APIs require the engine DocumentStore" }, 501); + return c.json( + { error: "retention APIs require the engine DocumentStore" }, + 501, + ); } try { const result = await deps.memory.setRetentionClass({ diff --git a/src/routes/search.ts b/src/routes/search.ts index 45b230e..6af4936 100644 --- a/src/routes/search.ts +++ b/src/routes/search.ts @@ -7,12 +7,7 @@ import { formatCaughtError, log } from "../log.js"; import { SearchRequest } from "../http-bodies.js"; import { MemoryError } from "../memory.js"; import type { RouteDeps } from "./deps.js"; -import { - caller, - grantGuard, - requirePrincipal, - resolveCaller, -} from "./deps.js"; +import { caller, grantGuard, requirePrincipal, resolveCaller } from "./deps.js"; // `kinds`/`entity_ids` scope every retrieval channel — see the // `kinds`/`entityIds` doc comments on MemorySearchParams (memory.ts) @@ -89,9 +84,7 @@ export function mountSearchRoute(app: Hono, deps: RouteDeps): void { ...(kinds !== undefined ? { kinds } : {}), ...(entity_ids !== undefined ? { entityIds: entity_ids } : {}), ...(sources !== undefined ? { sources } : {}), - ...(includeDeprecated !== undefined - ? { includeDeprecated } - : {}), + ...(includeDeprecated !== undefined ? { includeDeprecated } : {}), }); return c.json(result); } catch (err) { diff --git a/src/services/capture.test.ts b/src/services/capture.test.ts index c4a8ee7..1c46224 100644 --- a/src/services/capture.test.ts +++ b/src/services/capture.test.ts @@ -60,10 +60,14 @@ describe("embedInsertedChunksWithConfig — degraded reason array (CL-6287)", () function untouchableRawSql(): RawSql { return { unsafe: () => { - throw new Error("rawSql.unsafe must not be called when embed is unconfigured"); + throw new Error( + "rawSql.unsafe must not be called when embed is unconfigured", + ); }, begin: () => { - throw new Error("rawSql.begin must not be called when embed is unconfigured"); + throw new Error( + "rawSql.begin must not be called when embed is unconfigured", + ); }, } as unknown as RawSql; } @@ -116,7 +120,7 @@ function openaiEmbeddingBody(initBody: unknown): unknown { } return { data: Array.from({ length: count }, () => ({ - embedding: new Array(BACKGROUND_TEST_DIMS).fill(0.1), + embedding: Array.from({ length: BACKGROUND_TEST_DIMS }, () => 0.1), })), }; } @@ -124,7 +128,9 @@ function openaiEmbeddingBody(initBody: unknown): unknown { function unreachableRawSql(): RawSql { return { unsafe: () => { - throw new Error("rawSql.unsafe must not be called on the synchronous capture path"); + throw new Error( + "rawSql.unsafe must not be called on the synchronous capture path", + ); }, } as unknown as RawSql; } @@ -181,7 +187,10 @@ describe("captureDocument defers embedding to the background (CL-8615)", () => { }, }), new Promise((_, reject) => - setTimeout(() => reject(new Error("captureDocument awaited the embedder")), 1000), + setTimeout( + () => reject(new Error("captureDocument awaited the embedder")), + 1000, + ), ), ]); @@ -202,7 +211,9 @@ describe("runBackgroundEmbedPass retries then sweeps pending (CL-8615)", () => { const flakyFetch = (async (_url: unknown, init?: { body?: unknown }) => { fetchCalls++; if (fetchCalls === 1) { - throw new Error("connect ECONNREFUSED 127.0.0.1:11434 (busy local ollama)"); + throw new Error( + "connect ECONNREFUSED 127.0.0.1:11434 (busy local ollama)", + ); } return jsonResponse(openaiEmbeddingBody(init?.body)); }) as unknown as typeof fetch; @@ -215,7 +226,9 @@ describe("runBackgroundEmbedPass retries then sweeps pending (CL-8615)", () => { if (text.includes("LEFT JOIN")) { pendingSelectCalls++; if (pendingSelectCalls === 1) { - return Promise.resolve([{ id: "chunk_pending", text: "pending text" }]); + return Promise.resolve([ + { id: "chunk_pending", text: "pending text" }, + ]); } return Promise.resolve([]); } @@ -261,7 +274,10 @@ describe("background embed pass honors the configured embed timeout (CL-8615)", const timeoutMs = 45000; const seenSignals: unknown[] = []; const timeoutArgs: number[] = []; - const observingFetch = (async (_url: unknown, init?: { body?: unknown; signal?: unknown }) => { + const observingFetch = (async ( + _url: unknown, + init?: { body?: unknown; signal?: unknown }, + ) => { seenSignals.push(init?.signal); return jsonResponse(openaiEmbeddingBody(init?.body)); }) as unknown as typeof fetch; @@ -324,7 +340,9 @@ describe("findPendingChunks skips tombstoned and non-live versions", () => { expect(pending).toEqual([{ id: "chunk_live", text: "keep me" }]); expect(queries).toHaveLength(1); const { sql, params } = queries[0]!; - expect(sql).toContain('INNER JOIN "memory"."version" v ON v.id = c.version_id'); + expect(sql).toContain( + 'INNER JOIN "memory"."version" v ON v.id = c.version_id', + ); expect(sql).toContain("v.status <> 'tombstoned'"); expect(sql).toContain("v.generation = $3"); expect(params).toEqual(["tenant-1", 100, "live"]); diff --git a/src/services/capture.ts b/src/services/capture.ts index 46ac08b..4a6dc5a 100644 --- a/src/services/capture.ts +++ b/src/services/capture.ts @@ -909,15 +909,23 @@ async function attemptBackgroundEmbedPass(args: { ); if (degraded.length > 0) return { degraded }; try { - await reembedPendingChunks(args.sql, args.tenantId, args.embedClientConfig, { - fetchImpl: args.fetchImpl, - limit: args.pendingLimit, - }); + await reembedPendingChunks( + args.sql, + args.tenantId, + args.embedClientConfig, + { + fetchImpl: args.fetchImpl, + limit: args.pendingLimit, + }, + ); } catch (err) { - log.warn(`capture: background pending sweep failed: ${formatCaughtError(err)}`, { - tenantId: args.tenantId, - error: formatCaughtError(err), - }); + log.warn( + `capture: background pending sweep failed: ${formatCaughtError(err)}`, + { + tenantId: args.tenantId, + error: formatCaughtError(err), + }, + ); } return { degraded }; } @@ -965,7 +973,9 @@ export async function runBackgroundEmbedPass( } const delay = BACKGROUND_EMBED_RETRY_DELAYS_MS[attempt - 1] ?? - BACKGROUND_EMBED_RETRY_DELAYS_MS[BACKGROUND_EMBED_RETRY_DELAYS_MS.length - 1]!; + BACKGROUND_EMBED_RETRY_DELAYS_MS[ + BACKGROUND_EMBED_RETRY_DELAYS_MS.length - 1 + ]!; await resolved.sleep(delay); } } @@ -1026,12 +1036,18 @@ export async function captureDocument( tenantId: input.tenantId, chunks: backgroundChunks, run: (chunks) => - runBackgroundEmbedPass(deps.sql, input.tenantId, chunks, embedClientConfig, { - fetchImpl: resolved.fetchImpl, - sleep: resolved.sleep, - maxAttempts: resolved.maxAttempts, - pendingLimit: resolved.pendingLimit, - }), + runBackgroundEmbedPass( + deps.sql, + input.tenantId, + chunks, + embedClientConfig, + { + fetchImpl: resolved.fetchImpl, + sleep: resolved.sleep, + maxAttempts: resolved.maxAttempts, + pendingLimit: resolved.pendingLimit, + }, + ), }), ); @@ -1053,7 +1069,10 @@ export async function deriveFromRawCapture( input: CaptureInput, rawCaptureId: string, generation: string, - derivation: { chunker?: AdaptAndPlanOptions["chunker"]; embed: EmbedClientConfig }, + derivation: { + chunker?: AdaptAndPlanOptions["chunker"]; + embed: EmbedClientConfig; + }, ): Promise { const plan = adaptAndPlan( input.document, diff --git a/src/services/feed.ts b/src/services/feed.ts index 3959410..e5cc38a 100644 --- a/src/services/feed.ts +++ b/src/services/feed.ts @@ -5,10 +5,7 @@ import { and, asc, eq, gt, isNull, ne, or, sql } from "drizzle-orm"; import type { Db } from "../db/client.js"; -import { - memoryDocument, - memoryVersion, -} from "../db/schema.js"; +import { memoryDocument, memoryVersion } from "../db/schema.js"; import { LIVE_GENERATION } from "../core/generation.js"; export const FEED_LIMIT_MIN = 1; @@ -55,10 +52,7 @@ export class FeedInputError extends Error { } } -export async function fetchFeed( - db: Db, - args: FeedArgs, -): Promise { +export async function fetchFeed(db: Db, args: FeedArgs): Promise { const after = Math.max(0, Math.floor(args.after ?? 0)); const limit = Math.min( FEED_LIMIT_MAX, @@ -101,10 +95,7 @@ export async function fetchFeed( createdByPrincipalId: memoryVersion.createdByPrincipalId, }) .from(memoryVersion) - .innerJoin( - memoryDocument, - eq(memoryDocument.id, memoryVersion.documentId), - ) + .innerJoin(memoryDocument, eq(memoryDocument.id, memoryVersion.documentId)) .where(and(...conditions)) .orderBy(asc(memoryVersion.feedSeq)) .limit(limit); diff --git a/src/services/search.test.ts b/src/services/search.test.ts index f1e52d8..f9faa1c 100644 --- a/src/services/search.test.ts +++ b/src/services/search.test.ts @@ -39,20 +39,27 @@ function candidate(overrides: Partial = {}): CandidateRow { describe("authorityWeightedScore", () => { it("scales relevance linearly, from unchanged at 0 to +50% at 1", () => { - const cases: Array<[relevance: number, authority: number, expected: number]> = [ + const cases: Array< + [relevance: number, authority: number, expected: number] + > = [ [1, 1, 1.5], [0.4, 0, 0.4], [1, 0.5, 1.25], ]; for (const [relevance, authority, expected] of cases) { - expect(authorityWeightedScore(relevance, authority)).toBeCloseTo(expected, 10); + expect(authorityWeightedScore(relevance, authority)).toBeCloseTo( + expected, + 10, + ); } }); }); describe("snippet", () => { it("trims, truncates with an ellipsis, and honors maxLen", () => { - const cases: Array<[input: string, maxLen: number | undefined, expected: string]> = [ + const cases: Array< + [input: string, maxLen: number | undefined, expected: string] + > = [ ["hello world", undefined, "hello world"], [" hello world ", undefined, "hello world"], ["a".repeat(300), undefined, `${"a".repeat(240)}…`], @@ -67,9 +74,24 @@ describe("snippet", () => { describe("dedupeCandidatesPerDocument", () => { it("keeps only the highest authority-weighted-scoring chunk per document", () => { const rows = [ - candidate({ chunkId: "c1", documentId: "doc_a", rank: 0.5, authority: 0.2 }), - candidate({ chunkId: "c2", documentId: "doc_a", rank: 0.9, authority: 0.1 }), - candidate({ chunkId: "c3", documentId: "doc_b", rank: 0.3, authority: 0.9 }), + candidate({ + chunkId: "c1", + documentId: "doc_a", + rank: 0.5, + authority: 0.2, + }), + candidate({ + chunkId: "c2", + documentId: "doc_a", + rank: 0.9, + authority: 0.1, + }), + candidate({ + chunkId: "c3", + documentId: "doc_b", + rank: 0.3, + authority: 0.9, + }), ]; const deduped = dedupeCandidatesPerDocument(rows); expect(deduped.map((r) => r.chunkId).sort()).toEqual(["c2", "c3"]); @@ -77,8 +99,18 @@ describe("dedupeCandidatesPerDocument", () => { it("sorts the deduped result by authority-weighted score descending", () => { const rows = [ - candidate({ chunkId: "low", documentId: "doc_low", rank: 0.1, authority: 0 }), - candidate({ chunkId: "high", documentId: "doc_high", rank: 0.8, authority: 1 }), + candidate({ + chunkId: "low", + documentId: "doc_low", + rank: 0.1, + authority: 0, + }), + candidate({ + chunkId: "high", + documentId: "doc_high", + rank: 0.8, + authority: 1, + }), ]; const deduped = dedupeCandidatesPerDocument(rows); expect(deduped[0]?.chunkId).toBe("high"); @@ -109,8 +141,18 @@ describe("dedupeCandidatesPerDocument", () => { it("ranks by raw fused score alone when applyAuthorityPrior is false, never double-applying authority", () => { const rows = [ - candidate({ chunkId: "low-rank-high-authority", documentId: "doc_a", rank: 0.4, authority: 1 }), - candidate({ chunkId: "high-rank-low-authority", documentId: "doc_b", rank: 0.6, authority: 0 }), + candidate({ + chunkId: "low-rank-high-authority", + documentId: "doc_a", + rank: 0.4, + authority: 1, + }), + candidate({ + chunkId: "high-rank-low-authority", + documentId: "doc_b", + rank: 0.6, + authority: 0, + }), ]; const deduped = dedupeCandidatesPerDocument(rows, false); expect(deduped[0]?.chunkId).toBe("high-rank-low-authority"); @@ -128,7 +170,12 @@ describe("deriveHybridEvidence", () => { top?: Top; expected: "strong" | "weak" | "none"; }> = [ - { rows: [], count: 0, top: { rerankScore: 0.9, authority: 0.9 }, expected: "none" }, + { + rows: [], + count: 0, + top: { rerankScore: 0.9, authority: 0.9 }, + expected: "none", + }, { rows: [candidate({ rank: 0.01, authority: 0.9 })], count: 1, @@ -250,15 +297,15 @@ describe("fetchDenseCandidates hnsw tuning", () => { unsafe: (sqlText: string) => { statements.push(sqlText); return Promise.resolve( - sqlText.includes('FROM "memory"."embed_model"') - ? [MODEL_ROW] - : [], + sqlText.includes('FROM "memory"."embed_model"') ? [MODEL_ROW] : [], ); }, begin: (cb: (t: FakeTx) => Promise) => cb(tx), }; return { - rawSql: rawSql as unknown as Parameters[0]["sql"], + rawSql: rawSql as unknown as Parameters< + typeof fetchDenseCandidates + >[0]["sql"], statements, savepointAttempts: () => savepointAttempts, }; @@ -267,7 +314,11 @@ describe("fetchDenseCandidates hnsw tuning", () => { function args(sql: Parameters[0]["sql"]) { return { sql, - embedClientConfig: { baseUrl: "https://embed.example.com", modelId: "m", apiStyle: "openai" as const }, + embedClientConfig: { + baseUrl: "https://embed.example.com", + modelId: "m", + apiStyle: "openai" as const, + }, fetchImpl: openaiEmbedFetch(), tenantId: "tenant-1", principalId: null, @@ -280,7 +331,9 @@ describe("fetchDenseCandidates hnsw tuning", () => { const fake = fakeRawSql(); await fetchDenseCandidates(args(fake.rawSql)); expect(fake.statements).toContain("SET LOCAL hnsw.ef_search = 250"); - expect(fake.statements).toContain("SET LOCAL hnsw.iterative_scan = 'relaxed_order'"); + expect(fake.statements).toContain( + "SET LOCAL hnsw.iterative_scan = 'relaxed_order'", + ); expect(fake.savepointAttempts()).toBe(1); // Support is cached per pool: the second call sets the GUC directly @@ -293,23 +346,32 @@ describe("fetchDenseCandidates hnsw tuning", () => { }); it("degrades to ef_search alone on pgvector < 0.8 and stops probing", async () => { - const unknownGuc = Object.assign(new Error("unrecognized configuration parameter"), { - code: "42704", - }); + const unknownGuc = Object.assign( + new Error("unrecognized configuration parameter"), + { + code: "42704", + }, + ); const fake = fakeRawSql(unknownGuc); const rows = await fetchDenseCandidates(args(fake.rawSql)); expect(rows).toEqual([]); expect(fake.statements).toContain("SET LOCAL hnsw.ef_search = 250"); - expect(fake.statements.filter((s) => s.includes("iterative_scan"))).toHaveLength(0); + expect( + fake.statements.filter((s) => s.includes("iterative_scan")), + ).toHaveLength(0); await fetchDenseCandidates(args(fake.rawSql)); expect(fake.savepointAttempts()).toBe(1); }); it("rethrows a non-42704 savepoint failure", async () => { - const fake = fakeRawSql(Object.assign(new Error("connection reset"), { code: "08006" })); - await expect(fetchDenseCandidates(args(fake.rawSql))).rejects.toThrow("connection reset"); + const fake = fakeRawSql( + Object.assign(new Error("connection reset"), { code: "08006" }), + ); + await expect(fetchDenseCandidates(args(fake.rawSql))).rejects.toThrow( + "connection reset", + ); }); }); @@ -426,15 +488,15 @@ describe("fetchDenseCandidates kind/entity filtering", () => { statements.push(sqlText); return Promise.resolve( // CL-5233 qualified the table — only the fully-qualified form matches. - sqlText.includes('FROM "memory"."embed_model"') - ? [MODEL_ROW] - : [], + sqlText.includes('FROM "memory"."embed_model"') ? [MODEL_ROW] : [], ); }, begin: (cb: (t: FakeTx) => Promise) => cb(tx), }; return { - rawSql: rawSql as unknown as Parameters[0]["sql"], + rawSql: rawSql as unknown as Parameters< + typeof fetchDenseCandidates + >[0]["sql"], statements, }; } @@ -609,9 +671,13 @@ describe("hybridSearch — embed unconfigured (CL-6287)", () => { where: () => builder, orderBy: () => builder, limit: () => builder, - then: (onFulfilled: (v: unknown[]) => unknown, onRejected?: (e: unknown) => unknown) => - rows().then(onFulfilled, onRejected), - catch: (onRejected: (e: unknown) => unknown) => rows().catch(onRejected), + // oxlint-disable-next-line unicorn/no-thenable -- fakes drizzle's thenable query builder + then: ( + onFulfilled: (v: unknown[]) => unknown, + onRejected?: (e: unknown) => unknown, + ) => rows().then(onFulfilled, onRejected), + catch: (onRejected: (e: unknown) => unknown) => + rows().catch(onRejected), }; return builder; } @@ -625,10 +691,14 @@ describe("hybridSearch — embed unconfigured (CL-6287)", () => { function untouchableRawSql(): RawSql { return { unsafe: () => { - throw new Error("rawSql.unsafe must not be called when embed is unconfigured"); + throw new Error( + "rawSql.unsafe must not be called when embed is unconfigured", + ); }, begin: () => { - throw new Error("rawSql.begin must not be called when embed is unconfigured"); + throw new Error( + "rawSql.begin must not be called when embed is unconfigured", + ); }, } as unknown as RawSql; } @@ -657,7 +727,9 @@ describe("hybridSearch — embed unconfigured (CL-6287)", () => { db: fakeDb([candidate()]), sql: untouchableRawSql(), config: unconfiguredEmbedConfig(), - fetchImpl: mock(() => Promise.reject(new Error("unreachable"))) as unknown as typeof fetch, + fetchImpl: mock(() => + Promise.reject(new Error("unreachable")), + ) as unknown as typeof fetch, }, { query: "hello", tenantId: "tenant-1", principalId: null }, ); @@ -675,7 +747,9 @@ describe("hybridSearch — embed unconfigured (CL-6287)", () => { rank: 0.8, }); const fetchImpl = mock(() => - Promise.reject(new Error("fetch must not be called when embed is unconfigured")), + Promise.reject( + new Error("fetch must not be called when embed is unconfigured"), + ), ); const result = await hybridSearch( diff --git a/src/services/search.ts b/src/services/search.ts index 50e3c42..87d444c 100644 --- a/src/services/search.ts +++ b/src/services/search.ts @@ -27,7 +27,6 @@ import { RerankConfigError, RerankQueryTooLongError, validateRerankConfig, - type RerankClientConfig, } from "../core/rerank-client.js"; import { mmrRerank, type MmrItem } from "../core/mmr.js"; import { recordDegrade } from "../core/degrade-metrics.js"; @@ -238,7 +237,9 @@ export function toHit( hit.provenance = row.provenance as NonNullable; } if (row.sourceClass !== undefined) { - hit.source_class = row.sourceClass as NonNullable; + hit.source_class = row.sourceClass as NonNullable< + SearchHit["source_class"] + >; } if (row.derivedFrom !== undefined) { hit.derived_from = row.derivedFrom; @@ -564,14 +565,8 @@ export async function fetchLexicalCandidates( sourceClass: memoryVersion.sourceClass, }) .from(memoryChunk) - .innerJoin( - memoryVersion, - eq(memoryChunk.versionId, memoryVersion.id), - ) - .innerJoin( - memoryDocument, - eq(memoryChunk.documentId, memoryDocument.id), - ) + .innerJoin(memoryVersion, eq(memoryChunk.versionId, memoryVersion.id)) + .innerJoin(memoryDocument, eq(memoryChunk.documentId, memoryDocument.id)) .where(and(...conditions)) .orderBy(desc(rankExpr), desc(memoryVersion.occurredAt)) .limit(overfetchLimit); @@ -607,7 +602,9 @@ const iterativeScanSupport = new WeakMap(); * the GUC hard max. Non-finite input falls back to the default. */ export function hnswEfSearch(overfetchLimit: number): number { - const limit = Number.isFinite(overfetchLimit) ? Math.floor(overfetchLimit) : 40; + const limit = Number.isFinite(overfetchLimit) + ? Math.floor(overfetchLimit) + : 40; return Math.max(40, Math.min(1000, limit)); } @@ -1018,8 +1015,10 @@ export async function hybridSearch( includeDeprecated, }); - const embedClientConfig = resolvedTuning?.embed ?? toEmbedClientConfig(config.embed); - const rerankConfig = resolvedTuning?.rerank ?? toRerankClientConfig(config.rerank); + const embedClientConfig = + resolvedTuning?.embed ?? toEmbedClientConfig(config.embed); + const rerankConfig = + resolvedTuning?.rerank ?? toRerankClientConfig(config.rerank); let denseRows: CandidateRow[] = []; let degraded: DegradeFlag[] | undefined; diff --git a/src/services/share-grants.test.ts b/src/services/share-grants.test.ts index e763027..e6202d5 100644 --- a/src/services/share-grants.test.ts +++ b/src/services/share-grants.test.ts @@ -29,9 +29,9 @@ describe("buildShareGrants", () => { }); expect(grants).toHaveLength(2); expect(grants.every((g) => g.resource === "memory.doc:kdoc_1")).toBe(true); - expect(grants.every((g) => g.action === "search" && g.effect === "allow")).toBe( - true, - ); + expect( + grants.every((g) => g.action === "search" && g.effect === "allow"), + ).toBe(true); expect(grants.map((g) => g.principalId).sort()).toEqual(["bob", "carol"]); expect(grants[0]?.conditions?.memoryShare).toEqual({ sharedBy: "alice", diff --git a/src/services/timeline.test.ts b/src/services/timeline.test.ts index 072568a..1ec3311 100644 --- a/src/services/timeline.test.ts +++ b/src/services/timeline.test.ts @@ -28,7 +28,11 @@ describe("filterTimelineRows (grant-tag access)", () => { it("allows the creator without grants on tags", async () => { const { events, withheld } = await filterTimelineRows( [row({ createdByPrincipalId: "p1", accessTags: [] })], - { principalId: "p1", tenantId: "t1", grants: createInMemoryGrantStore([]) }, + { + principalId: "p1", + tenantId: "t1", + grants: createInMemoryGrantStore([]), + }, ); expect(withheld).toBe(0); expect(events).toHaveLength(1); @@ -37,7 +41,12 @@ describe("filterTimelineRows (grant-tag access)", () => { it("denies a peer without a matching grant", async () => { const { events, withheld } = await filterTimelineRows( - [row({ createdByPrincipalId: "owner", accessTags: ["memory.tenant:t1"] })], + [ + row({ + createdByPrincipalId: "owner", + accessTags: ["memory.tenant:t1"], + }), + ], { principalId: "peer", tenantId: "t1", @@ -79,7 +88,11 @@ describe("filterTimelineRows (grant-tag access)", () => { const { events, withheld } = await filterTimelineRows( [ row({ documentId: "a", createdByPrincipalId: "p1", title: "mine" }), - row({ documentId: "b", createdByPrincipalId: "other", title: "theirs" }), + row({ + documentId: "b", + createdByPrincipalId: "other", + title: "theirs", + }), ], { principalId: "p1", tenantId: "t1" }, ); diff --git a/src/services/timeline.ts b/src/services/timeline.ts index 3f3a967..b8cf118 100644 --- a/src/services/timeline.ts +++ b/src/services/timeline.ts @@ -7,7 +7,7 @@ * * See docs/AUTHZ-DOCUMENT-ACCESS.md. */ -import { and, desc, eq, sql } from "drizzle-orm"; +import { and, desc, eq } from "drizzle-orm"; import type { ConditionRegistry, GrantStore } from "@intx/authz"; import { canAccessDocument, matchesVisibleTags } from "../grant-tags.js"; import { LIVE_GENERATION } from "../core/generation.js"; @@ -133,11 +133,11 @@ export async function filterTimelineRows( export async function listTimelineEvents( params: ListTimelineParams, ): Promise { - const limit = Math.min( - Math.max(params.limit ?? DEFAULT_LIMIT, 1), - MAX_LIMIT, + const limit = Math.min(Math.max(params.limit ?? DEFAULT_LIMIT, 1), MAX_LIMIT); + const fetchLimit = Math.min( + limit * TIMELINE_OVERFETCH, + MAX_LIMIT * TIMELINE_OVERFETCH, ); - const fetchLimit = Math.min(limit * TIMELINE_OVERFETCH, MAX_LIMIT * TIMELINE_OVERFETCH); const generation = params.generation ?? LIVE_GENERATION; const rows = await params.db diff --git a/src/services/transform.test.ts b/src/services/transform.test.ts index 1a0c3e9..bcc5d32 100644 --- a/src/services/transform.test.ts +++ b/src/services/transform.test.ts @@ -19,7 +19,9 @@ describe("buildRerankClientConfig", () => { }); it("defaults apiStyle to 'tei' when omitted, mirroring the engine's own rerank config precedent", () => { - const config = buildRerankClientConfig({ baseUrl: "https://rerank.example" }); + const config = buildRerankClientConfig({ + baseUrl: "https://rerank.example", + }); expect(config).toEqual({ baseUrl: "https://rerank.example", apiStyle: "tei", @@ -52,7 +54,11 @@ describe("TransformConfigParamsSchema", () => { it("accepts a fully-specified params object", () => { const parsed = TransformConfigParamsSchema({ chunk: { strategy: "token.recursive", maxTokens: 500 }, - embed: { baseUrl: "http://localhost:11434", model: "nomic-embed-text", apiStyle: "ollama" }, + embed: { + baseUrl: "http://localhost:11434", + model: "nomic-embed-text", + apiStyle: "ollama", + }, rerank: { baseUrl: "https://rerank.example", apiStyle: "tei" }, authorityWeight: 0.8, recencyHalfLifeDays: 45, @@ -65,7 +71,11 @@ describe("TransformConfigParamsSchema", () => { it("rejects an unknown chunk strategy — token.recursive is the only one adaptAndPlan supports", () => { const parsed = TransformConfigParamsSchema({ chunk: { strategy: "semantic.experimental" }, - embed: { baseUrl: "http://localhost:11434", model: "nomic-embed-text", apiStyle: "ollama" }, + embed: { + baseUrl: "http://localhost:11434", + model: "nomic-embed-text", + apiStyle: "ollama", + }, }); expect(parsed instanceof type.errors).toBe(true); }); @@ -159,10 +169,14 @@ describe("runTransform / promoteGeneration — embed-absent guards (CL-6287)", ( function untouchableRawSql(): RawSql { return { unsafe: () => { - throw new Error("rawSql.unsafe must not be called when embed is unconfigured"); + throw new Error( + "rawSql.unsafe must not be called when embed is unconfigured", + ); }, begin: () => { - throw new Error("rawSql.begin must not be called when embed is unconfigured"); + throw new Error( + "rawSql.begin must not be called when embed is unconfigured", + ); }, } as unknown as RawSql; } @@ -182,8 +196,10 @@ describe("runTransform / promoteGeneration — embed-absent guards (CL-6287)", ( function chain(table: unknown) { const resolve = (): Promise => { - if (table === transformConfig) return Promise.resolve(rows.transformConfig ?? []); - if (table === transformRun) return Promise.resolve(rows.transformRun ?? []); + if (table === transformConfig) + return Promise.resolve(rows.transformConfig ?? []); + if (table === transformRun) + return Promise.resolve(rows.transformRun ?? []); if (table === rawCapture) return Promise.resolve(rows.rawCapture ?? []); return Promise.resolve([]); }; @@ -199,11 +215,13 @@ describe("runTransform / promoteGeneration — embed-absent guards (CL-6287)", ( updates.push(v); return builder; }, + // oxlint-disable-next-line unicorn/no-thenable -- fakes drizzle's thenable query builder then: ( onFulfilled: (v: unknown[]) => unknown, onRejected?: (e: unknown) => unknown, ) => resolve().then(onFulfilled, onRejected), - catch: (onRejected: (e: unknown) => unknown) => resolve().catch(onRejected), + catch: (onRejected: (e: unknown) => unknown) => + resolve().catch(onRejected), }; return builder; } @@ -235,7 +253,12 @@ describe("runTransform / promoteGeneration — embed-absent guards (CL-6287)", ( expect(result.status).toBe("failed"); expect(updates).toHaveLength(1); - const update = updates[0] as { status: string; rawCount: number; versionCount: number; error: string | null }; + const update = updates[0] as { + status: string; + rawCount: number; + versionCount: number; + error: string | null; + }; expect(update.status).toBe("failed"); expect(update.rawCount).toBe(0); expect(update.versionCount).toBe(0); diff --git a/src/services/transform.ts b/src/services/transform.ts index f2fe99f..7d7007b 100644 --- a/src/services/transform.ts +++ b/src/services/transform.ts @@ -4,11 +4,7 @@ import type { Db, RawSql } from "../db/client.js"; import type { EngineConfig } from "../config.js"; import { newId } from "../core/id.js"; import { formatCaughtError, log } from "../log.js"; -import { - rawCapture, - transformConfig, - transformRun, -} from "../db/schema.js"; +import { rawCapture, transformConfig, transformRun } from "../db/schema.js"; import { TransformConfigParamsSchema, type TransformConfigParams, @@ -18,11 +14,19 @@ import { import { AdaptedDocumentSchema } from "../core/schemas/adapted-document.js"; import { chunkTokenRecursive } from "../core/chunk/token-recursive.js"; import type { Chunker } from "../core/chunk/types.js"; -import { EmbedClientConfigSchema, type EmbedClientConfig } from "../core/embed-client.js"; +import { + EmbedClientConfigSchema, + type EmbedClientConfig, +} from "../core/embed-client.js"; import type { RerankClientConfig } from "../core/rerank-client.js"; import { deriveFromRawCapture, type CaptureInput } from "./capture.js"; import { LIVE_GENERATION } from "../core/generation.js"; -import { activateEmbedModel, activateEmbedModelByKey, clearActiveEmbedModels, resolveActiveEmbedTable } from "../core/embed-model-registry.js"; +import { + activateEmbedModel, + activateEmbedModelByKey, + clearActiveEmbedModels, + resolveActiveEmbedTable, +} from "../core/embed-model-registry.js"; import { createRawSqlClient } from "../core/embed-sql.js"; export class TransformConfigNotFoundError extends Error { @@ -185,8 +189,12 @@ async function loadTransformRun( function buildChunker(params: TransformConfigParams["chunk"]): Chunker { return (text) => chunkTokenRecursive(text, { - ...(params.maxTokens !== undefined ? { maxTokens: params.maxTokens } : {}), - ...(params.minTokens !== undefined ? { minTokens: params.minTokens } : {}), + ...(params.maxTokens !== undefined + ? { maxTokens: params.maxTokens } + : {}), + ...(params.minTokens !== undefined + ? { minTokens: params.minTokens } + : {}), ...(params.overlapTokens !== undefined ? { overlapTokens: params.overlapTokens } : {}), @@ -418,7 +426,11 @@ export async function runTransform( let totalRows = 0; try { - const rawRows = await selectRawCaptureRows(deps.db, configRow.tenantId, scope); + const rawRows = await selectRawCaptureRows( + deps.db, + configRow.tenantId, + scope, + ); totalRows = rawRows.length; const chunker = buildChunker(configRow.params.chunk); // A replay re-derives (and re-embeds) a corpus; it makes no sense @@ -429,7 +441,10 @@ export async function runTransform( "transform run requires an embed endpoint (EMBED_BASE_URL/EMBED_MODEL) — none is configured on this engine", ); } - const embed = buildEmbedClientConfig(configRow.params.embed, deps.config.embed); + const embed = buildEmbedClientConfig( + configRow.params.embed, + deps.config.embed, + ); for (const row of rawRows) { try { @@ -523,7 +538,9 @@ export async function promoteGeneration( input: { tenantId: string; generation: string }, ): Promise { if (input.generation === LIVE_GENERATION) { - throw new TransformPromoteError("cannot promote the live generation onto itself"); + throw new TransformPromoteError( + "cannot promote the live generation onto itself", + ); } const runRows = await deps.db @@ -566,7 +583,10 @@ export async function promoteGeneration( "cannot promote generation: no embed endpoint is configured on this engine", ); } - const embed = buildEmbedClientConfig(configRow.params.embed, deps.config.embed); + const embed = buildEmbedClientConfig( + configRow.params.embed, + deps.config.embed, + ); const archiveGen = `archive_${run.id}_${Date.now()}`; const readClient = createRawSqlClient(deps.sql); diff --git a/src/sidecar-bundle.test.ts b/src/sidecar-bundle.test.ts index 9f8668e..88e96f1 100644 --- a/src/sidecar-bundle.test.ts +++ b/src/sidecar-bundle.test.ts @@ -1,7 +1,11 @@ import { describe, expect, test } from "bun:test"; import { MEMORY_TOOL_DEFINITIONS } from "./tools.js"; -import { HUB_CREDENTIAL_HANDLE, memory, SIDECAR_BUNDLE_ID } from "./sidecar-bundle.js"; +import { + HUB_CREDENTIAL_HANDLE, + memory, + SIDECAR_BUNDLE_ID, +} from "./sidecar-bundle.js"; type Recorded = { url: string; init?: RequestInit }; @@ -9,7 +13,10 @@ function env(recorded: Recorded[], respond: () => Response) { const credential = { kind: "http" as const, fetch: (input: string | URL | Request, init?: RequestInit) => { - recorded.push({ url: String(input), ...(init !== undefined ? { init } : {}) }); + recorded.push({ + url: String(input), + ...(init !== undefined ? { init } : {}), + }); return Promise.resolve(respond()); }, dispose: () => undefined, @@ -18,7 +25,8 @@ function env(recorded: Recorded[], respond: () => Response) { address: "run-1@acme.example.com", capabilities: { resolve: (key: string) => { - if (key !== "credentials") throw new Error(`unexpected capability ${key}`); + if (key !== "credentials") + throw new Error(`unexpected capability ${key}`); return { resolve: (handle: string) => { if (handle !== HUB_CREDENTIAL_HANDLE) { @@ -33,10 +41,13 @@ function env(recorded: Recorded[], respond: () => Response) { } const ok = () => - new Response(JSON.stringify({ data: { documentId: "doc_1", versionId: "ver_1" } }), { - status: 200, - headers: { "content-type": "application/json" }, - }); + new Response( + JSON.stringify({ data: { documentId: "doc_1", versionId: "ver_1" } }), + { + status: 200, + headers: { "content-type": "application/json" }, + }, + ); const signal = new AbortController().signal; @@ -104,7 +115,10 @@ describe("the memory sidecar bundle", () => { test("an unknown tool name is a tool error", async () => { const recorded: Recorded[] = []; const bundle = memory(env(recorded, ok)); - const result = await bundle.run({ id: "d", name: "memory_nope", arguments: {} }, signal); + const result = await bundle.run( + { id: "d", name: "memory_nope", arguments: {} }, + signal, + ); expect(result.isError).toBe(true); expect(recorded).toHaveLength(0); }); @@ -126,7 +140,10 @@ describe("every declared tool maps onto a route", () => { test(`${name} calls ${url}`, async () => { const recorded: Recorded[] = []; const bundle = memory(env(recorded, ok)); - const result = await bundle.run({ id: name, name, arguments: args }, signal); + const result = await bundle.run( + { id: name, name, arguments: args }, + signal, + ); expect(result.isError).toBeUndefined(); expect(recorded[0]?.url).toBe(url); await bundle.dispose?.(); @@ -154,7 +171,11 @@ describe("every declared tool maps onto a route", () => { const recorded: Recorded[] = []; const bundle = memory(env(recorded, ok)); await bundle.run( - { id: "n", name: "@corbits/memory/sidecar-bundle:memory_list", arguments: {} }, + { + id: "n", + name: "@corbits/memory/sidecar-bundle:memory_list", + arguments: {}, + }, signal, ); expect(recorded[0]?.url).toBe("/api/workflow-memory/list"); @@ -162,7 +183,10 @@ describe("every declared tool maps onto a route", () => { test("a body-less hub error still reads as a tool error", async () => { const bundle = memory(env([], () => new Response("nope", { status: 500 }))); - const result = await bundle.run({ id: "e", name: "memory_list", arguments: {} }, signal); + const result = await bundle.run( + { id: "e", name: "memory_list", arguments: {} }, + signal, + ); expect(result.isError).toBe(true); expect(result.content).toBe("the hub answered 500"); }); diff --git a/src/sidecar-bundle.ts b/src/sidecar-bundle.ts index e3e2057..aecc2f5 100644 --- a/src/sidecar-bundle.ts +++ b/src/sidecar-bundle.ts @@ -37,7 +37,9 @@ type HttpCredential = { /** The slice of the host-assembled runtime capabilities this bundle uses. */ type CredentialCapabilities = { - resolve(key: "credentials"): { resolve(handle: string): Promise }; + resolve(key: "credentials"): { + resolve(handle: string): Promise; + }; }; /** The env keys `requires` declares, on top of the core ones. */ @@ -62,9 +64,7 @@ function query(params: Record): string { return serialized === "" ? "" : `?${serialized}`; } -function defined( - entries: Record, -): Record { +function defined(entries: Record): Record { return Object.fromEntries( Object.entries(entries).filter(([, value]) => value !== undefined), ); @@ -72,7 +72,10 @@ function defined( /** Maps one model-facing tool call onto the run-scoped route that performs * it. An unknown name returns undefined and answers as a tool error. */ -function requestFor(name: string, args: Record): Request_ | undefined { +function requestFor( + name: string, + args: Record, +): Request_ | undefined { switch (name) { case "memory_add": return { @@ -150,14 +153,21 @@ export async function callMemoryRoute( runAddress: string, request: Request_, ): Promise { - const response = await fetchImpl(`${WORKFLOW_MEMORY_BASE_PATH}${request.path}`, { - method: request.method, - headers: { - "x-workflow-run-address": runAddress, - ...(request.body !== undefined ? { "content-type": "application/json" } : {}), + const response = await fetchImpl( + `${WORKFLOW_MEMORY_BASE_PATH}${request.path}`, + { + method: request.method, + headers: { + "x-workflow-run-address": runAddress, + ...(request.body !== undefined + ? { "content-type": "application/json" } + : {}), + }, + ...(request.body !== undefined + ? { body: JSON.stringify(request.body) } + : {}), }, - ...(request.body !== undefined ? { body: JSON.stringify(request.body) } : {}), - }); + ); if (!response.ok) throw new Error(await readErrorMessage(response)); const payload: unknown = await response.json().catch(() => undefined); if (typeof payload === "object" && payload !== null && "data" in payload) { @@ -202,7 +212,11 @@ const bundleOpts = { const name = bareToolName(call.name); const request = requestFor(name, call.arguments); if (request === undefined) { - return { callId: call.id, content: `unknown memory tool: ${call.name}`, isError: true }; + return { + callId: call.id, + content: `unknown memory tool: ${call.name}`, + isError: true, + }; } try { const credential = await hub(); diff --git a/src/tools.ts b/src/tools.ts index 31ed825..bf2c845 100644 --- a/src/tools.ts +++ b/src/tools.ts @@ -4,7 +4,12 @@ * stays free of any agent-SDK dependency, and `sidecar-bundle.ts` binds each * name to the run-scoped route that performs it. */ -import { LIST_LIMIT_MAX, LIST_LIMIT_MIN, SEARCH_LIMIT_MAX, SEARCH_LIMIT_MIN } from "./limits.js"; +import { + LIST_LIMIT_MAX, + LIST_LIMIT_MIN, + SEARCH_LIMIT_MAX, + SEARCH_LIMIT_MIN, +} from "./limits.js"; export type MemoryToolDefinition = { name: string; @@ -48,7 +53,8 @@ export const MEMORY_TOOL_DEFINITIONS: readonly MemoryToolDefinition[] = [ provenance: { type: "string", enum: ["stated", "inferred", "unknown"], - description: "How content was obtained (inferred for distilled claims)", + description: + "How content was obtained (inferred for distilled claims)", }, lineage_class: { type: "string", @@ -65,8 +71,14 @@ export const MEMORY_TOOL_DEFINITIONS: readonly MemoryToolDefinition[] = [ items: { type: "string" }, description: "Source version ids this claim is derived from", }, - valid_from: { type: "string", description: "Optional validity start (ISO)" }, - valid_until: { type: "string", description: "Optional validity end (ISO)" }, + valid_from: { + type: "string", + description: "Optional validity start (ISO)", + }, + valid_until: { + type: "string", + description: "Optional validity end (ISO)", + }, share: { type: "object", properties: { @@ -114,7 +126,8 @@ export const MEMORY_TOOL_DEFINITIONS: readonly MemoryToolDefinition[] = [ sources: { type: "array", items: { type: "string" }, - description: 'Optional channel filter (e.g. "local" and/or live source ids)', + description: + 'Optional channel filter (e.g. "local" and/or live source ids)', }, includeEvidence: { type: "boolean", @@ -131,7 +144,8 @@ export const MEMORY_TOOL_DEFINITIONS: readonly MemoryToolDefinition[] = [ }, { name: "memory_list", - description: "List recent documents in your workbench's memory, including your teammates'.", + description: + "List recent documents in your workbench's memory, including your teammates'.", inputSchema: { type: "object", properties: { @@ -168,7 +182,8 @@ export const MEMORY_TOOL_DEFINITIONS: readonly MemoryToolDefinition[] = [ }, exclude_generator: { type: "string", - description: "Skip versions written by this generator_agent_id (loop-safety)", + description: + "Skip versions written by this generator_agent_id (loop-safety)", }, }, additionalProperties: false, diff --git a/src/workflow-mount.test.ts b/src/workflow-mount.test.ts index 7a02046..47dab88 100644 --- a/src/workflow-mount.test.ts +++ b/src/workflow-mount.test.ts @@ -48,7 +48,9 @@ function fakeMemory(seen: Record[]): Memory { } as unknown as Memory; } -function agentTokenAuth(overrides: Partial = {}): AgentTokenAuth { +function agentTokenAuth( + overrides: Partial = {}, +): AgentTokenAuth { return { verify: (ctx) => { const c = ctx as { req: { header(name: string): string | undefined } }; @@ -61,7 +63,10 @@ function agentTokenAuth(overrides: Partial = {}): AgentTokenAuth }; } -function host(seen: Record[], agentToken: AgentTokenAuth = agentTokenAuth()) { +function host( + seen: Record[], + agentToken: AgentTokenAuth = agentTokenAuth(), +) { return mountWorkflowMemory(new Hono(), { memory: fakeMemory(seen), agentToken, @@ -73,7 +78,11 @@ describe("agent-token authentication", () => { const seen: Record[] = []; const res = await host(seen).request("/list", { headers: agentHeaders }); expect(res.status).toBe(200); - expect(seen[0]).toMatchObject({ verb: "list", tenantId: "acme", principalId: "agent-1" }); + expect(seen[0]).toMatchObject({ + verb: "list", + tenantId: "acme", + principalId: "agent-1", + }); }); test("no bearer at all is refused", async () => { @@ -89,7 +98,9 @@ describe("agent-token authentication", () => { const seen: Record[] = []; const app = host( seen, - agentTokenAuth({ verify: () => ({ tenantId: "other", definitionId: "def-1" }) }), + agentTokenAuth({ + verify: () => ({ tenantId: "other", definitionId: "def-1" }), + }), ); const res = await app.request("/list", { headers: agentHeaders }); expect(res.status).toBe(401); @@ -99,7 +110,10 @@ describe("agent-token authentication", () => { test("an address that names no run is refused", async () => { const seen: Record[] = []; const res = await host(seen).request("/list", { - headers: { ...agentHeaders, "x-workflow-run-address": "run-9@acme.example.com" }, + headers: { + ...agentHeaders, + "x-workflow-run-address": "run-9@acme.example.com", + }, }); expect(res.status).toBe(401); expect(seen).toHaveLength(0); @@ -131,7 +145,12 @@ describe("the routes the memory tools call", () => { body: JSON.stringify({ query: "q", limit: 3, entity_ids: ["e1"] }), }); expect(res.status).toBe(200); - expect(seen[0]).toMatchObject({ verb: "search", query: "q", limit: 3, entityIds: ["e1"] }); + expect(seen[0]).toMatchObject({ + verb: "search", + query: "q", + limit: 3, + entityIds: ["e1"], + }); }); test("feed forwards the cursor and generator exclusion", async () => { diff --git a/src/workflow-mount.ts b/src/workflow-mount.ts index 85764c6..3da60f4 100644 --- a/src/workflow-mount.ts +++ b/src/workflow-mount.ts @@ -62,7 +62,10 @@ export type AgentTokenAuth = { ) => Promise | AgentTokenIdentity | undefined; resolveRun: ( runAddress: string, - ) => Promise | ResolvedWorkflowRunScope | null; + ) => + | Promise + | ResolvedWorkflowRunScope + | null; }; export type WorkflowMemoryEnv = { @@ -85,14 +88,22 @@ export function mountWorkflowMemory( ): Hono { const { memory, agentToken } = opts; - const authenticate: MiddlewareHandler = async (c, next) => { + const authenticate: MiddlewareHandler = async ( + c, + next, + ) => { const identity = await agentToken.verify(c); const address = c.req.header("x-workflow-run-address") ?? ""; - const scope = identity === undefined ? null : await agentToken.resolveRun(address); + const scope = + identity === undefined ? null : await agentToken.resolveRun(address); // Same 401 whether the bearer was unrecognized, the address named no run, // or the run belongs to another tenant: a bearer learns nothing from the // difference. - if (identity === undefined || scope === null || scope.tenantId !== identity.tenantId) { + if ( + identity === undefined || + scope === null || + scope.tenantId !== identity.tenantId + ) { return c.json( { error: "Missing or unrecognized bearer token / run address" }, 401, @@ -104,7 +115,11 @@ export function mountWorkflowMemory( }; app.use("*", authenticate); - function failure(c: Parameters>[0], verb: string, err: unknown) { + function failure( + c: Parameters>[0], + verb: string, + err: unknown, + ) { if (err instanceof MemoryError) { return c.json({ error: err.message }, err.status as 400 | 501); } @@ -121,7 +136,8 @@ export function mountWorkflowMemory( return c.json({ error: "Invalid JSON body" }, 400); } const body = AddRequest(raw); - if (body instanceof type.errors) return c.json({ error: body.summary }, 400); + if (body instanceof type.errors) + return c.json({ error: body.summary }, 400); const scope = c.get("workflowRunScope"); try { @@ -129,7 +145,9 @@ export function mountWorkflowMemory( content: { title: body.title, text: body.text }, tenantId: scope.tenantId, principalId: scope.principalId, - ...(body.access_tags !== undefined ? { accessTags: body.access_tags } : {}), + ...(body.access_tags !== undefined + ? { accessTags: body.access_tags } + : {}), // A workbench is the team: a run's memories are shared with it by // default, and an explicit share only ever adds to that. share: { ...((body.share ?? {}) as ShareSugar), tenant: true }, @@ -137,12 +155,24 @@ export function mountWorkflowMemory( ...(body.generator_agent_id !== undefined ? { generatorAgentId: body.generator_agent_id } : {}), - ...(body.provenance !== undefined ? { provenance: body.provenance } : {}), - ...(body.lineage_class !== undefined ? { lineageClass: body.lineage_class } : {}), - ...(body.temporal_class !== undefined ? { temporalClass: body.temporal_class } : {}), - ...(body.derived_from !== undefined ? { derivedFrom: body.derived_from } : {}), - ...(body.valid_from !== undefined ? { validFrom: body.valid_from } : {}), - ...(body.valid_until !== undefined ? { validUntil: body.valid_until } : {}), + ...(body.provenance !== undefined + ? { provenance: body.provenance } + : {}), + ...(body.lineage_class !== undefined + ? { lineageClass: body.lineage_class } + : {}), + ...(body.temporal_class !== undefined + ? { temporalClass: body.temporal_class } + : {}), + ...(body.derived_from !== undefined + ? { derivedFrom: body.derived_from } + : {}), + ...(body.valid_from !== undefined + ? { validFrom: body.valid_from } + : {}), + ...(body.valid_until !== undefined + ? { validUntil: body.valid_until } + : {}), }); return c.json({ data: result }); } catch (err) { @@ -158,7 +188,8 @@ export function mountWorkflowMemory( return c.json({ error: "Invalid JSON body" }, 400); } const body = SearchRequest(raw); - if (body instanceof type.errors) return c.json({ error: body.summary }, 400); + if (body instanceof type.errors) + return c.json({ error: body.summary }, 400); const scope = c.get("workflowRunScope"); try { @@ -169,7 +200,9 @@ export function mountWorkflowMemory( visibleTags: teamTags(scope), ...(body.limit !== undefined ? { limit: body.limit } : {}), ...(body.kinds !== undefined ? { kinds: body.kinds } : {}), - ...(body.entity_ids !== undefined ? { entityIds: body.entity_ids } : {}), + ...(body.entity_ids !== undefined + ? { entityIds: body.entity_ids } + : {}), ...(body.sources !== undefined ? { sources: body.sources } : {}), ...(body.includeEvidence !== undefined ? { includeEvidence: body.includeEvidence } @@ -204,7 +237,10 @@ export function mountWorkflowMemory( app.get("/feed", async (c) => { if (memory.feed === undefined) { - return c.json({ error: "feed is not available on this memory plane" }, 501); + return c.json( + { error: "feed is not available on this memory plane" }, + 501, + ); } const after = c.req.query("after"); const limit = c.req.query("limit"); @@ -212,7 +248,9 @@ export function mountWorkflowMemory( const parsed = parseFeedQuery({ ...(after !== undefined ? { after } : {}), ...(limit !== undefined ? { limit } : {}), - ...(excludeGenerator !== undefined ? { exclude_generator: excludeGenerator } : {}), + ...(excludeGenerator !== undefined + ? { exclude_generator: excludeGenerator } + : {}), }); if (!parsed.ok) return c.json({ error: parsed.error }, 400);