diff --git a/.corbits/review-round2-bruckheimer.md b/.corbits/review-round2-bruckheimer.md deleted file mode 100644 index 1e7e6e0..0000000 --- a/.corbits/review-round2-bruckheimer.md +++ /dev/null @@ -1,41 +0,0 @@ -# Bruckheimer — PR #31 round 2 (HEAD `6189b56`) - -## One-liner - -Foundation so a company-brain distiller can write **inferred claims** with -provenance, rank them in time, re-distill offline without trashing live search, -and share docs with real grants — not just tags. - -## Hook progress - -| Piece | Status for the hook | -|-------|---------------------| -| Claim-bearing / derived_from / provenance | Shipped | -| Temporal classes + validity | Shipped | -| Staged transform / promote / demote | Shipped (code); weak tests | -| Share grants materialization | Shipped (fail-soft without writable store) | -| Capture feed / distiller workflow | **Not this PR** (CL-5868/5869) | -| Schema/config host-friendly | DATABASE_URL + memory schema — good | - -## Product blockers - -1. **Dense entity filter broken after rename** — if any host/search UI filters by - entity, dense path dies. Fix before merge. -2. **Promote rollback untested** — if demote is the safety valve for bad - distillation, untested demote is a product risk, not just eng debt. -3. **Share without WritableGrantStore** only warns — hosts will think share - “worked.” Consider fail-loud option or return receipt with - `grantsMaterialized: false` (follow-up OK). - -## Breaking-change cost/benefit - -- `knowledge` → `memory` schema: right name for the product; cost is fresh - install only. Acceptable if no production tenants on old schema. -- `DATABASE_URL` preferred: lowers host friction (one DB). Risk: host with two - URLs silently picks the wrong one — document clearly (done in mount-config). - -## Ship advice - -Fix the dense SQL bug, clarify CHANGELOG on versionId, merge as foundation. -Do not hold the PR for the distiller itself. Next product milestone is feed + -workflow, not more schema polish. diff --git a/.corbits/review-round2-convergence.md b/.corbits/review-round2-convergence.md deleted file mode 100644 index dc84bb3..0000000 --- a/.corbits/review-round2-convergence.md +++ /dev/null @@ -1,88 +0,0 @@ -# Convergence review — PR #31 round 2 (HEAD `6189b56`) - -**Method:** 8 deep lenses in-session (fleet spawn blocked: Codex profile -`fleur` unauthorized). Lenses: critique, greybeard, gaasbot, neckbeard, -bruckheimer, security, schema, OSS. Artifacts: -`.corbits/review-round2-*.md`. - -## Converged verdict - -**CHANGES REQUESTED — one hard blocker, then merge-eligible.** - -All eight lenses agree the distillation foundation is the right shape and that -prior promote/demote/versionId work improved the bar. All eight that touched -search/schema flag the same ship-blocker. - ---- - -## Must-fix before merge (unanimous / multi-lens) - -| ID | Finding | Lenses | Action | -|----|---------|--------|--------| -| **M1** | Dense entity filter SQL still uses `knowledge_edge`; table is `"memory"."edge"` (`src/services/search.ts:537`) | critique, greybeard, gaasbot, neckbeard, security, schema, OSS, bruckheimer | Fix SQL + add regression test (dense path with entityIds) | -| **M2** | CHANGELOG Unreleased “Previously” claims add returns only `{ documentId }`; code returns `versionId` | critique, gaasbot, greybeard, OSS | Correct CHANGELOG | - -## Should-fix (same PR if cheap; else ticket) - -| ID | Finding | Lenses | Action | -|----|---------|--------|--------| -| **S1** | No promote/demote service E2E regression | critique, greybeard, gaasbot, bruckheimer | Add transform test: ensure→promote→demote restores model_key + generation | -| **S2** | search.test mocks still accept `FROM knowledge_embed_model` | neckbeard, OSS | Match only `"memory"."embed_model"` | -| **S3** | Document that transform/promote/demote are host-privileged (no principal on API, no HTTP) | security, gaasbot, greybeard | Short IMPLEMENTATION / AUTHZ note | - -## Follow-up (do not block merge) - -| ID | Finding | Lenses | -|----|---------|--------| -| F1 | `setActiveEmbedModelExclusive` two-step race | critique, security | -| F2 | Promote/demote multi-step windows; consider tenant advisory lock | critique, greybeard, security | -| F3 | Share fail-soft without WritableGrantStore / appendAccessTags — receipt field | critique, bruckheimer, security | -| F4 | TS `knowledge*` identifiers + migration filenames + comment rot | neckbeard, schema | -| F5 | Ops note for `ALTER SCHEMA knowledge RENAME TO memory` if any old install | greybeard, schema | -| F6 | README advanced surface (transform exports) | OSS, bruckheimer | - -## Explicit non-goals this PR - -- Distiller workflow / capture feed (CL-5868/5869) -- HTTP routes for transform -- Bulk rename of knowledge* TypeScript symbols -- Re-introducing separate knowledge DB - -## Steelman of “merge as-is” - -Tests are green; entityIds on dense may be rare; rename is fresh-install-only. -**Rejected:** a single untested raw-SQL island after a schema rename is exactly -the class of bug that survives CI and fails first real use. Fix is trivial. - -## Steelman of “hold for promote E2E” - -Demote is the safety valve for bad distillation. **Partial accept:** S1 is -high value but not a correctness hole in the current code path (demote restore -exists). Prefer same-PR if <1h; else ticket linked from PR. - -## Converged fix order - -1. **M1** fix + test -2. **M2** CHANGELOG -3. **S2** tighten mocks (with M1 test) -4. **S1** if time -5. **S3** one paragraph -6. Re-run typecheck + test → push - -## Bar after fixes - -| Bar | Status after M1+M2 | -|-----|--------------------| -| Security | Pass (with S3 doc preferred) | -| Product | Pass foundation | -| Greybeard / architecture | Pass | -| OSS quality | Pass pre-1.0 | -| Critique | Pass with S1 follow-up | - ---- - -## Note on fleet - -All 8 `task` spawns failed: `Codex profile "fleur" is not authorized`. Reviews -were executed in-session with the same multi-lens briefs. Re-auth `/model` -(profile fleur) to restore sub-agent fleet for future rounds. diff --git a/.corbits/review-round2-critique.md b/.corbits/review-round2-critique.md deleted file mode 100644 index 7e4ae77..0000000 --- a/.corbits/review-round2-critique.md +++ /dev/null @@ -1,74 +0,0 @@ -# Critique — PR #31 round 2 (HEAD `6189b56`) - -In-session (fleet blocked: Codex profile `fleur`). Diff `origin/main..HEAD`. -Typecheck clean; 367 tests green at last gate. - -## Verdict - -**CHANGES REQUESTED.** Prior promote/demote and versionId fixes landed, but the -`knowledge` → `memory` rename left a **live dense-path SQL bug**, and -promote/demote still lack end-to-end regression tests. - -## Critical - -1. **Dense entity filter still queries `knowledge_edge`** - (`src/services/search.ts:537`). After schema rename, the table is - `"memory"."edge"`. Any dense search with `entityIds` will fail at runtime - (`relation "knowledge_edge" does not exist`). Lexical path is fine (Drizzle - `knowledgeEdge` → `memory.edge`). No test exercises dense+entityIds. - -## High - -2. **No automated promote → demote → dense restore E2E.** Registry unit tests - cover ensure/activate/byKey, but nothing asserts: staged run under model B - does not flip live active; promote swaps generation + activates staged; - demote restores generation **and** prior model_key. CL-5872 rollback - acceptance still untested at the service layer. - -3. **Promote/demote are multi-step outside a single transaction.** Activate - dense, then version swap (or reverse on demote). Concurrent promote of two - generations, or crash mid-window, can leave dense target and generation tags - briefly inconsistent. Documented preference is intentional; still a - production footgun without locks / single-flight. - -## Medium - -4. **`setActiveEmbedModelExclusive` is two non-atomic UPDATEs** - (`embed-model-registry.ts:309-325`). Concurrent activate of A and B can - leave two `active` rows until next exclusive call; `ORDER BY updated_at` - picks one, but window exists. - -5. **Transform plane methods take only `tenantId` / `configId` — no principal.** - In-process API; no HTTP routes. Correct for library shape, but any host that - re-exports without its own grant check hands promote/demote to any caller - who can reach the plane. Docs should state “host must authorize.” - -6. **Share path still fail-soft** when `appendAccessTags` or WritableGrantStore - missing (`memory.ts:707-727`): warns, continues, peers fail-closed. Easy to - miss in production. - -7. **CHANGELOG drift:** Unreleased “Previously” still says `add` returns - `{ documentId }` only; wire now returns `{ documentId, versionId }`. - -## Low / Nits - -8. Comments and enums still say `knowledge.version` / `knowledge.embed_model` - (`enums.ts`, `embed-model-registry.ts:34`, `generation.ts`). -9. Migration file still named `0002_knowledge_baseline.sql` while creating - `memory.*`. -10. TS exports still `knowledgeDocument` / `knowledgeVersion` under memory schema. - -## Test gaps - -- Dense search + `entityIds` (would catch Critical #1). -- `promoteGeneration` / `demoteGeneration` service tests with fake SQL + version rows. -- Concurrent exclusive activate (optional stress). -- `loadMemoryConfig` DATABASE_URL vs KNOWLEDGE preference already covered. - -## Assumptions challenged - -- “Rename was complete because migrations and drizzle use memory” — raw SQL - island in dense path was not. -- “367 green ⇒ rename safe” — unit tests mock embed_model with dual match - (`knowledge_embed_model` OR `memory.embed_model`) and never hit entity filter - raw SQL. diff --git a/.corbits/review-round2-gaasbot.md b/.corbits/review-round2-gaasbot.md deleted file mode 100644 index c1d1735..0000000 --- a/.corbits/review-round2-gaasbot.md +++ /dev/null @@ -1,44 +0,0 @@ -# Gaasbot (CTO) — PR #31 round 2 (HEAD `6189b56`) - -## CTO verdict - -**Right foundation, one ship-blocker.** This is the correct shape for -resident distillation: claim-bearing + temporal + staged transform + grants. -Do not expand scope into the distiller workflow (CL-5869) on this PR. - -## Must-fix-before-merge - -1. Fix dense-path `knowledge_edge` → `"memory"."edge"` (`search.ts:537`). -2. Add a regression test that would fail on that bug (dense fetch with - entityIds, assert SQL contains `"memory"."edge"` or run against real SQL - mock that only knows memory.edge). -3. Fix CHANGELOG: add returns `versionId`; remove contradictory “Previously” - line or mark superseded. - -## Can-ship-with-followups - -- Promote/demote service-level tests. -- Tenant-scoped advisory lock on promote/demote. -- Atomic exclusive activate (single SQL CTE or transaction). -- Explicit “host authorizes transform APIs” note in IMPLEMENTATION.md. -- Rename TS `knowledge*` symbols in a dedicated PR (not this one). - -## Defer - -- HTTP routes for transform/promote (in-process is fine for v1 distiller). -- Capture feed (CL-5868), relevancy (CL-5867), retention (CL-5871). -- Upgrade migration from `knowledge` schema for old DBs unless a customer exists. - -## Architecture notes - -- Exporting transform + embed registry from package root is aggressive but OK - for the distiller as first-party consumer. Keep them off HTTP until grants - exist. -- Preferring `DATABASE_URL` is correct for “same Postgres, own schema.” Warn - hosts that still set both URLs with different values — preferred wins. -- Do not re-introduce a separate knowledge DB requirement. - -## Priority - -Blocker fix is a one-liner + test. Merge after that; iterate on promote -hardening in the same branch if cheap, else follow-up ticket. diff --git a/.corbits/review-round2-greybeard.md b/.corbits/review-round2-greybeard.md deleted file mode 100644 index 7af7025..0000000 --- a/.corbits/review-round2-greybeard.md +++ /dev/null @@ -1,59 +0,0 @@ -# Greybeard — PR #31 round 2 (HEAD `6189b56`) - -## Verdict - -**HOLD for one correctness fix; then ship foundation.** Architecture of -ensure-vs-activate, claim-bearing, temporal classes, and share materialization -is sound. Schema rename + DATABASE_URL is the right long-term shape. - -## Ship / hold - -**Hold** until dense `entityIds` SQL is fixed (`knowledge_edge` → `"memory"."edge"`). -After that: **ship with follow-ups** (promote E2E tests, exclusive activate -transaction, host authz docs for transform). - -## Critical / High - -1. **Raw SQL residue after schema rename** — `search.ts:537` `knowledge_edge`. - Irreversible-looking renames that leave one path broken are worse than no - rename: green CI + red prod. - -2. **Fresh-install-only schema rename** is honest in CHANGELOG but operationally - harsh. Acceptable for pre-1.0 / no prod tenants; document a one-shot - `ALTER SCHEMA knowledge RENAME TO memory` + table renames for anyone who - already migrated under `knowledge`. - -## Medium — design debt (acceptable for now) - -3. **JS identifiers lag schema** (`knowledgeDocument` table → `document`). Fine - if intentional transitional; pick a rename PR later — do not half-rename. -4. **Embedding tables keyed only by model_key**, multi-tenant rows inside. - Tenant filter on every dense query is load-bearing; keep that invariant in - review checklist forever. -5. **Promote activate-then-swap** preference is documented and reasonable. - Prefer advisory lock per tenant around promote/demote before multi-tenant - production load. -6. **Docs generally lockstep** with DATABASE_URL / memory schema after last - commit; IMPLEMENTATION table and AGENTS.md match. CHANGELOG “Previously” - still contradicts versionId on add. - -## Doc drift - -| Claim | Reality | -|-------|---------| -| CHANGELOG: add returns `{ documentId }` | Returns `{ documentId, versionId }` | -| Comments: knowledge.embed_model | Table is memory.embed_model | -| open.type "memory" | Correct in search.ts | - -## Design decisions that aged well - -- `ensureEmbedModel` vs `activateEmbedModel` split (replay must not steal live). -- `activateEmbedModelByKey` for demote without re-probe. -- `archived_live_model_key` on transform_run. -- Share grants + pass-through condition registry. -- Enum lockstep test (enums.lockstep.test.ts). - -## Recommendation - -Fix Critical SQL → add dense+entityIds test → optional promote E2E → merge. -Do not block on knowledge* TypeScript renames. diff --git a/.corbits/review-round2-neckbeard.md b/.corbits/review-round2-neckbeard.md deleted file mode 100644 index ff03a4b..0000000 --- a/.corbits/review-round2-neckbeard.md +++ /dev/null @@ -1,32 +0,0 @@ -# Neckbeard — PR #31 round 2 (HEAD `6189b56`) - -## Real bugs hiding as nits - -1. **`knowledge_edge` in raw SQL** (`search.ts:537`) — not a naming nit. **Bug.** -2. **search.test.ts still accepts `FROM knowledge_embed_model`** as a success - path for mocks (`search.test.ts:237, 412`). Teaches the wrong table name; - should only match `"memory"."embed_model"`. - -## Naming debt inventory (cosmetic unless noted) - -| Residue | Severity | -|---------|----------| -| `knowledgeDocument`, `knowledgeVersion`, `knowledgeChunk`, `knowledgeEdge`, `knowledgeEntity`, `knowledgeEmbedModel` exports | Cosmetic / API-internal | -| `KNOWLEDGE_SCHEMA` deprecated alias | OK transitional | -| `migrations/0002_knowledge_baseline.sql` filename | Cosmetic; content correct | -| Comments `knowledge.version`, `knowledge.embed_model` | Doc rot | -| Id prefixes `kver`, `kdoc` | Cosmetic | -| grant-tags test still uses `knowledge.project:ke` as a free-form tag | Fine (host tags) | -| CHANGELOG “Postgres schema name remains knowledge” removed; good | — | - -## Nits - -- Dual match in tests for old embed_model table should die with the rename. -- `### Previously` in CHANGELOG is nonstandard Keep-a-Changelog structure. -- Package still says “knowledge plane” in a few comments (`config.ts` FTS). -- `openTarget` comment still says “generic knowledge doc” (`search.ts:156`). - -## What not to rewrite - -Do not rename all `knowledge*` TS symbols in this PR. Ship the SQL fix and -stop. A bulk rename PR with codemod is fine later. diff --git a/.corbits/review-round2-oss.md b/.corbits/review-round2-oss.md deleted file mode 100644 index 890f2a1..0000000 --- a/.corbits/review-round2-oss.md +++ /dev/null @@ -1,53 +0,0 @@ -# OSS / public API quality — PR #31 round 2 (HEAD `6189b56`) - -## Public surface inventory (package root) - -- `createMemory`, `loadMemoryConfig`, `runMemoryMigrations`, `MemoryError` -- Routes: `registerMemoryRoutes` -- Ports: DocumentStore types, fakes, WritableGrantStore -- Share: buildShareGrants, materializeShareGrants, MEMORY_SHARE_* -- Transform: createTransformConfig, runTransform, promote/demoteGeneration, … -- Embed registry: ensure/activate/resolve helpers -- Degrade metrics, FTS helpers - -**Note:** Transform + embed registry on the root export is a large surface for -an “add/search/list” product blurb. Acceptable for distiller-as-consumer; -README should mention advanced APIs. - -## Breaking changes completeness - -| Change | CHANGELOG | Code | -|--------|-----------|------| -| Schema knowledge → memory | Yes | Yes | -| DATABASE_URL preferred | Yes | Yes | -| open.type memory | Yes | Yes | -| add returns versionId | **Stale “Previously” says no** | Yes | -| Claim-bearing / temporal / transform | Partial (IMPLEMENTATION) | Yes | - -## Quality bar - -| Area | Pass? | Notes | -|------|-------|-------| -| arktype at edges | Pass | transform params, raw_capture replay | -| Enum lockstep tests | Pass | enums.lockstep.test.ts | -| Tenant SQL discipline | Pass* | *except broken edge table name | -| Module focus | Pass | services split reasonably | -| Docs match exports | Partial | CHANGELOG versionId; comments knowledge.* | -| Test coverage public contracts | Partial | no promote E2E; dense+entityIds missing | -| Semver honesty | Partial | fix Unreleased Previously section | - -## Must-fix for OSS merge - -1. Dense entity SQL table name. -2. CHANGELOG honesty on `MemoryAddResult` / wire body. -3. Regression test for (1). - -## Nice-to-have before wider publish - -- README section: transform/promote privileged, host-gated. -- Drop dual mock match for `knowledge_embed_model` in tests. -- Do not bulk-rename knowledge* TS identifiers in this PR. - -## Verdict - -**Fail OSS bar until Critical SQL + CHANGELOG; then pass for pre-1.0 foundation.** diff --git a/.corbits/review-round2-schema.md b/.corbits/review-round2-schema.md deleted file mode 100644 index d8bb692..0000000 --- a/.corbits/review-round2-schema.md +++ /dev/null @@ -1,60 +0,0 @@ -# Schema / migrations — PR #31 round 2 (HEAD `6189b56`) - -## Summary - -Migrations and Drizzle schema consistently use `"memory".…` with short table -names (`document`, `version`, `chunk`, `edge`, …). Config prefers DATABASE_URL. -One application raw-SQL path still names the pre-rename edge table. - -## Critical - -1. **App SQL vs migration mismatch:** `search.ts:537` uses `knowledge_edge`; - migrations create `"memory"."edge"`. Dense entity filter is broken on any - real Postgres. - -## High - -2. **No upgrade path** from prior `knowledge` schema installs — CHANGELOG says - fresh-only. Correct if intentional; add a short ops note (RENAME SCHEMA + - RENAME tables) if anyone already applied old branch migrations. - -3. **Migration filenames** still `0002_knowledge_baseline.sql` etc. Content is - memory.*; confusing for operators grepping filenames. Optional rename of - files is risky if migration ledger already records names — leave filenames, - fix comments at top of 0002. - -## Medium - -4. **`0001_extensions.sql`** must create schema `memory` before 0002 — verify - CREATE SCHEMA IF NOT EXISTS memory (assumed present; was part of rename). -5. **`archived_live_model_key`** text, no FK to embed_model — intentional - (model row may be demoted/deleted); demote fails closed if key missing. -6. **embed_model ON CONFLICT** updates dims/model_id without status change — - correct for ensure. -7. **CHECK/arktype lockstep** covered by `enums.lockstep.test.ts` — good. -8. **Internal FKs** document ← version ← chunk; edge/entity free of control- - plane FKs — matches AGENTS.md. -9. **DATABASE_URL resolution** order correct; tests cover prefer / fallback / - throw. - -## Low - -10. Index names dropped `knowledge_` prefix — good. -11. Dynamic embedding tables: FK to memory.chunk; tenant_id column; no per- - tenant table isolation (by design). - -## Config - -| Source | Behavior | -|--------|----------| -| `memory.databaseUrl` on config | Programmatic hosts | -| `DATABASE_URL` | Preferred env | -| `KNOWLEDGE_DATABASE_URL` | Deprecated alias | -| Neither | throw | - -## Ranked defects - -1. Critical: raw SQL `knowledge_edge` -2. High: document upgrade path or confirm zero external installs -3. Medium: 0002 file header still says “knowledge plane” -4. Low: TS knowledge* symbols / comment rot diff --git a/.corbits/review-round2-security.md b/.corbits/review-round2-security.md deleted file mode 100644 index 7c9124c..0000000 --- a/.corbits/review-round2-security.md +++ /dev/null @@ -1,72 +0,0 @@ -# Security — PR #31 round 2 (HEAD `6189b56`) - -## Summary - -Tenant isolation on SQL paths is generally solid (tenant_id first; grant post- -filter). No auth in package (by design). One correctness issue can cause -hard-fail (DoS of entity-filtered dense search). Transform APIs are privileged -operations without built-in principal checks. - -## Critical - -None for classic IDOR/authz bypass found in this pass. Closest: - -**C-adjacent:** Dense `entityIds` query references non-existent `knowledge_edge` -(`search.ts:537`) — availability/DoS of that code path, not data leak. - -## High - -1. **Transform promote/demote/run lack principal binding** - (`memory.ts:751-798`, `transform.ts`). Anyone who can call the in-process - plane with a tenantId can rewrite that tenant’s live generation and active - embed model. Mitigation: host-only, no HTTP. **Requirement:** document that - hosts must gate these like admin APIs; never expose unauthenticated. - -2. **Shared embedding physical tables across tenants** (table name = - `embedding_` only). Isolation is `WHERE tenant_id = $1` on every - dense query. A missing tenant predicate on a future query is cross-tenant - vector leak. Current dense SQL includes `e.tenant_id = $1 AND c.tenant_id = $1`. - **Keep as permanent review invariant.** - -## Medium - -3. **Exclusive activate race** — two concurrent activates can briefly leave two - active rows; resolve picks latest updated_at. Unlikely privilege issue; - wrong model for search is integrity issue. - -4. **Share grants use `origin: "system"`** (`share-grants.ts:92`) with - always-true condition evaluator. Correct for not fail-closing, but grants - look “system-minted.” Audit trail relies on `conditions.memoryShare` payload. - Ensure host UIs show that payload. - -5. **Promote activate-before-swap window** — dense points at new model while - live versions still old (or reverse on demote). Transient wrong hits, not - cross-tenant. - -6. **`appendAccessTags` optional** — if missing, peer grants may not match tags; - peers fail-closed (safe) but sharer believes share succeeded. - -## Low - -7. Model endpoint URLs trusted by design (AGENTS.md) — no SSRF filter. Host - responsibility. -8. Dynamic SQL for embed table names validated by `EMBED_TABLE_NAME_PATTERN` / - modelKey hex — good. -9. Dynamic `dims` interpolated only after integer bounds check — good. - -## Recommended fixes - -| # | Fix | -|---|-----| -| 1 | `"memory"."edge"` in dense entity SQL + test | -| 2 | Doc: transform APIs are privileged; host grant required | -| 3 | Optional: single-transaction exclusive activate | -| 4 | Optional: return share materialization receipt to caller | - -## Attack scenarios checked (no exploit) - -- Cross-tenant document via search without grants → post-filter + tenant SQL. -- Embed table name injection → pattern reject. -- Promote another tenant’s generation → tenantId filter on transform_run; - config tenant mismatch check on promote. -- Share grant self-grant → skipped when peer === sharedBy. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 38c1b13..752d48c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,46 +1,37 @@ -name: CI +name: ci on: + pull_request: push: branches: [main] - pull_request: jobs: - test: + check: runs-on: ubuntu-latest + timeout-minutes: 15 services: postgres: image: pgvector/pgvector:pg17 - env: - POSTGRES_PASSWORD: memory-test - ports: - - 5432:5432 + env: { POSTGRES_PASSWORD: postgres, POSTGRES_DB: memory } + ports: ["5432:5432"] options: >- - --health-cmd pg_isready --health-interval 5s --health-timeout 5s - --health-retries 10 + --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 env: - TEST_DATABASE_URL: postgres://postgres:memory-test@localhost:5432/postgres + TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/memory steps: - - uses: actions/checkout@v4 - - uses: oven-sh/setup-bun@v2 + - uses: actions/checkout@v5 + - uses: actions/setup-node@v4 with: - bun-version: latest + node-version: 24 + - uses: oven-sh/setup-bun@v2 - run: bun install --frozen-lockfile - - run: bun run lint - - run: bun run format:check - - run: bun run typecheck - - run: bun run test + - run: bun run check - run: bun run test:e2e - node: - runs-on: ubuntu-latest - strategy: - matrix: - node-version: [24] - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: ${{ matrix.node-version }} - - run: npm install --no-audit --no-fund - - run: npx tsc --noEmit - - run: npm pack --dry-run + - name: node consumer smoke + run: | + set -euo pipefail + TARBALL="$PWD/$(npm pack --silent)" + mkdir -p "$RUNNER_TEMP/c" && cd "$RUNNER_TEMP/c" + npm init -y >/dev/null && npm pkg set type=module >/dev/null + npm install "$TARBALL" + node -e 'import("@corbits/memory").then((m) => { for (const n of ["createMemory", "createMemoryRoutes", "loadMemoryConfig", "mountWorkflowMemory", "createMemoryHttpClient"]) if (typeof m[n] !== "function") throw new Error("missing export: " + n); })' diff --git a/.github/workflows/cla.yml b/.github/workflows/cla.yml new file mode 100644 index 0000000..a03ad39 --- /dev/null +++ b/.github/workflows/cla.yml @@ -0,0 +1,53 @@ +name: CLA Assistant + +on: + issue_comment: + types: [created] + pull_request_target: + types: [opened, synchronize] + +permissions: + actions: write + contents: write + pull-requests: write + statuses: write + +concurrency: + group: cla-${{ github.event.pull_request.number || github.event.issue.number || github.run_id }} + cancel-in-progress: true + +jobs: + cla: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Maintainer fast path + if: >- + github.event_name == 'pull_request_target' && + (contains(fromJSON('["TheGreatAxios","brianjfox"]'), github.event.pull_request.user.login) || + endsWith(github.event.pull_request.user.login, '[bot]')) + run: echo "Maintainer or bot pull request; CLA not required." + - name: CLA Assistant + if: >- + ((github.event.comment.body == 'recreate-signatures' || + github.event.comment.body == 'I have read the CLA Document and I hereby sign the CLA') || + github.event_name == 'pull_request_target') && + !(github.event_name == 'pull_request_target' && + (contains(fromJSON('["TheGreatAxios","brianjfox"]'), github.event.pull_request.user.login) || + endsWith(github.event.pull_request.user.login, '[bot]'))) + # corbitsdev/cla-assistant-action v2.6.1-node24: upstream v2.6.1 on node24. + uses: corbitsdev/cla-assistant-action@ef6d3e51db8232fe93090810f13bde30497c1d74 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + with: + path-to-document: "https://github.com/${{ github.repository }}/blob/main/CLA.md" + path-to-signatures: "signatures/version1/cla.json" + branch: "cla-signatures" + allowlist: TheGreatAxios,brianjfox,*[bot] + custom-notsigned-prcomment: >- + Thank you for your contribution. Before it can be merged, please read our + [Contributor License Agreement](https://github.com/${{ github.repository }}/blob/main/CLA.md) + and sign it by posting a new comment on this pull request containing + exactly the line below (nothing else): + custom-pr-sign-comment: "I have read the CLA Document and I hereby sign the CLA" + custom-allsigned-prcomment: "All contributors have signed the CLA." diff --git a/.gitignore b/.gitignore index 8dc025e..6f17218 100644 --- a/.gitignore +++ b/.gitignore @@ -1,23 +1,8 @@ node_modules/ dist/ +*.tsbuildinfo +*.tgz +coverage/ .env -.env.local -*.log +.env.* .DS_Store -data/ -coverage/ - -# Local product-hub runtime state (docker volumes cover Postgres; these are file-based) -.hub-data/ -.agent-state/ -tmp/ - -# Dispatch orchestration scratch (local only) -dispatch/ - -# Staging dirs for sibling package extracts (copy out with cp only) -.staging-*/ - - -# Local agent/session scratch — never commit -.corbits/ diff --git a/AGENTS.md b/AGENTS.md index 385b2ae..e3e3beb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,25 +1,17 @@ -# Agent guide — @corbits/memory +# AGENTS.md -A library, not a service. `src/` is the whole product: a memory **add / search / -list** SDK that **mounts onto a host Interchange app**. There is no server, -port, or process entrypoint here, and there never should be. +## Purpose -## Commands - -```bash -bun install # Bun 1.2+ required -bun run typecheck # tsc --noEmit -bun run test # bun test ./src (no network, no Postgres) -bun run db:setup # apply migrations (needs DATABASE_URL) -docker compose up -d # local pgvector + model endpoints for manual runs -``` - -CI runs `typecheck` + `test` — both must pass before any push. +`@corbits/memory` is a memory add / search / list SDK that mounts onto a host +Interchange app. It owns the `memory` Postgres schema (pgvector), capture, +hybrid search, retention and the optional distiller helpers. It does not own +authentication, grants, tenancy, or model inference: embedding and reranking +are outbound HTTP calls to configured endpoints. There is no server, port or +process entrypoint here, and there never should be. ## Layout - `src/index.ts` — public surface: `createMemory`, `createMemoryRoutes` - - `src/mount-config.ts` / `src/config.ts` — mount config + engine config - `src/routes/` — Hono routes (`add`, `search`, `list`, `feed`, retention `forget`/`purge`/`retention-class`) @@ -32,11 +24,12 @@ CI runs `typecheck` + `test` — both must pass before any push. - `src/ports/` — `DocumentStore` / `SourceProvider` + fakes - `src/core/` — embed/rerank clients, merge, arktype schemas - `src/db/` + `migrations/` — Drizzle schema + SQL migrations (pgvector, `memory.*`) -- `packages/` — removed; DocumentStore adapters and Linear tools are sibling packages - (`@corbits/mem0-memory-adapter`, `@corbits/supermemory-memory-adapter`, - `@corbits/linear-tools`). +- `src/distiller/` — optional process helpers (`@corbits/memory/distiller`) +- `docs/` — how access control, temporal ranking, relevancy, retention and the + feed work; referenced from the code +- `e2e/` — real-Postgres suites -## Non-negotiable invariants +## Rules 1. **Authenticate nothing.** Identity defaults to `c.get("principal")` from the Interchange context; a host may instead supply `callerResolver` @@ -72,11 +65,8 @@ CI runs `typecheck` + `test` — both must pass before any push. responses. Keep the version range compatible with Interchange's catalog (currently `^2.1.29`). -## Docs - -- `PRODUCT.md` — what this is and what is out of scope (read first) -- `ARCHITECTURE.md` — why an SDK, design decisions -- `IMPLEMENTATION.md` — env vars, data model, service internals +## Local development -Keep all three current when behavior changes (`/scribe` maintains them). -License is LGPL-2.1 (`LICENSE`); contributions go through `CLA.md`. +```sh +bun install && bun run check +``` diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md deleted file mode 100644 index f6c36b3..0000000 --- a/ARCHITECTURE.md +++ /dev/null @@ -1,166 +0,0 @@ -# Corbits Memory — Architecture - -A memory **add / search / list** SDK that mounts onto an Interchange hub. The -host owns auth, tenancy, and the process; this library owns the durable memory -plane and the protected routes that read and write it. - -## Why an SDK, not a service - -The store was detachable from a larger backend, then mountable: - -- No memory table has a foreign key into any control-plane table — cross-refs - (`tenant_id`, `principal_id`, source refs) are plain `text`. Tables live in - the **`memory`** schema (same Postgres URL as the host is fine). -- Embedding and reranking go out as plain HTTP to configured model endpoints. -- Document access is Interchange grant tags on the row (`accessTags` + creator), - not a private ACL engine inside this package. - -It ships as `createMemory(…)` plus `createMemoryRoutes(deps)`: the host builds -the plane, mounts the returned Hono sub-app, and passes its `requireGrant`; the -routes read identity from request context and talk to the DocumentStore. No -second server. - -## Product path - -``` -add → ingest elements (store/chunk/embed) → process (optional, host) -``` - -``` -tools / host ingest workflow → /api/tenants/:tenantId/memory/* → Memory plane → DocumentStore - ↑ - Interchange auth + principal + grants - -sidecar-bundle (deployed agent) → /api/workflow-memory/* → Memory plane → DocumentStore - ↑ - hub credential + x-workflow-run-address (mountWorkflowMemory, no session) -``` - -Mount is intentionally small. The host already has `app`, grants, and -principal middleware; memory only needs to be handed those and the vector -config (or an injected store). - -**Ingest elements** run on the default store inside `add` (raw capture, chunks, -edges, embed). **Process** (claims, LLM link/classify) is host-owned inference, -preferably in the same workflow body as the add. Capture **feed** + distiller -helpers are optional multi-writer / backfill — not the primary path. - -## Boundaries - -- **Runtime**: Bun + Hono, mounted on the host app. **DB**: own pgvector - Postgres (`DATABASE_URL`) unless `documentStore` is injected. - **Types**: arktype at every route boundary. -- **No auth of its own.** By default, Interchange resolves the caller and - puts `principal` + `tenant` on context; routes read identity from there - (`tenantId = principal.tenantId`, `principalId = principal.id`). A host - with a non-browser caller (e.g. a workflow-run child with its own sidecar - bearer token) may instead pass `callerResolver` to `createMemoryRoutes` - — the host still does 100% of the authenticating, it just - hands the resolved `{ tenantId, principalId }` in through the seam instead - of setting context itself. Either way the resolved identity, never - anything from the request body, is what `grantGuard` authorizes. -- **Grants delegate to the host.** Pass `grantStore` + `conditionRegistry`; - routes use `createRequireGrant("memory", action)`. -- **Two authorization mechanisms, not one — know which is source of truth - for what.** (1) Grant tags decide _capability_ (may this principal call - `add`/`search`/`forget`/`purge` at all — `requireGrant`) and _visibility_ - (which documents a principal may see — `accessTags` + `canAccessDocument` - in `grant-tags.ts`, where a share grant legitimately widens who can find a - document). (2) A separate, imperative **ownership** check — the creator - lookup in `services/retention-ownership.ts`, called from `memory.ts` — - decides who may _forget or purge_ a specific document, and is the sole - source of truth for "whose document is this": it is never derived from - grant tags and a share grant never satisfies it. `MemoryGrantRequirement. -installHint` (`grant-requirements.ts`) looks adjacent to this but is not: - it is advisory metadata for install tooling sizing a capability grant, - read by nothing at request time. Do not extend mechanism (1) expecting it - to cover ownership — extend `retention-ownership.ts` instead. -- **Dependencies**: `@intx/hub-api`, `@intx/authz`, `@intx/log`, Hono, Drizzle, - arktype, `postgres`, `hono-openapi`. LGPL-2.1 — see `LICENSE`. - -## Identity — context in, data out - -1. **Who is calling** is the request principal. Clients never send - `tenant_id` / `principal_id` on the body. -2. **What is stored** is opaque data: `tenant_id`, `principal_id`, - `created_by_kind`, `access_tags`, source refs. Queries scope by `tenant_id` - first; document access is grant tags + creator. - -## Layers (default pgvector store) - -- `raw_capture` — immutable original content (replay substrate). -- `derived` — chunks / embeddings / authority / edges from raw. -- `transform_config` + replay — rebuild derived from raw without re-fetch. - -Injected DocumentStores own their own persistence model; the plane still -exposes the same three verbs. - -## Mounted surface - -`createMemoryRoutes(deps)`, mounted at `/api/tenants/:tenantId/memory`, serves: - -- `POST /api/tenants/:tenantId/memory/add` — ingest (raw + derive on the default store). -- `POST /api/tenants/:tenantId/memory/search` — hybrid retrieval (FTS + dense → RRF → rerank → - authority/recency → MMR); optional live `SourceProvider` merge (fail-soft). -- `GET /api/tenants/:tenantId/memory/list` — recent documents, same grant-tag filter as local - search. -- `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` — tombstone - (grant `memory:forget`; creator-only, see below). -- `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` — hard - delete (grant `memory:purge`; creator-only; irreversible). -- `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` — - set retention class (grant `memory:forget`; creator-only). - -Forget and purge are deliberately separate routes and separate grant actions -(never one route with a boolean flag) — a host wiring a "forget this" button -cannot accidentally wire up permanent deletion. `sweepEphemeral` (TTL -auto-deprecation) is **not** HTTP-routed: it is a maintenance sweep a host -schedules on its own cron, not a user action; call it in-process against the -returned `Memory`. See docs/RETENTION.md. - -Returns an in-process `Memory` (`add`, `search`, `list`, `close`, plus the -optional retention writes) for host workers and ingestion modules that -already resolved identity. - -**Capture** is the write path inside `add` (raw capture → chunks / edges / -embed on the default store). **Search** is hybrid retrieval on the same -plane, whether the caller arrived via tenant routes or the sidecar mount. - -## Sidecar mount (`mountWorkflowMemory`) - -Deployed agents do not install this package as a git sidecar. They carry -the factory at `@corbits/memory/sidecar-bundle`, which holds no client -code, no base URL, and no token: it resolves the host `hub` credential and -calls the run-scoped routes under `/api/workflow-memory/*`. That mount is -**parallel** to the tenant routes — two auth conventions stay on two -mounts so neither is harder to reason about. - -```ts -import { mountWorkflowMemory } from "@corbits/memory"; - -mountWorkflowMemory(workflowMemoryApp, { - memory, - agentToken: { verify, resolveRun }, -}); -app.route("/api/workflow-memory", workflowMemoryApp); -``` - -Authorization on this mount **is the token itself**: the hub only mints an -agent token for a definition it already authorized, and every call is -confined to the verified run's tenant and principal. The mount runs **no -grant check of its own** and has no tenant override. A bearer minted for -one workbench cannot act on another's run (`verify` tenant must match -`resolveRun` tenant). Unrecognized bearer, unknown run address, and -cross-tenant mismatch all return the same **401**. - -The sidecar factory (`src/sidecar-bundle.ts`) maps `memory_add` / -`memory_search` / `memory_list` / `memory_feed` onto those run-scoped -routes. Relative paths only — a mediated HTTP handle resolves them against -the origin it is pinned to. Capture and search still execute on the same -in-process `Memory` plane as the tenant routes. - -## Provenance - -Framework-agnostic core (chunking, embed/rerank clients, hybrid search, MMR) -was extracted from an internal RAG implementation. Persistence and the -mountable surface are native to this repo. diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index c82e898..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,53 +0,0 @@ -# Changelog - -All notable changes to `@corbits/memory` are documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## [0.2.0] — 2026-09-25 - -### Changed - -- **Breaking:** `@intx/*`, `drizzle-orm`, `hono`, `hono-openapi` and - `postgres` are peer dependencies; the host supplies them. `engines` is - removed. -- **Breaking:** `createMemoryRoutes({ memory, requireGrant, callerResolver? })` - returns the memory routes as a `Hono` sub-app with paths relative - to its mount point; hosts mount it at `/api/tenants/:tenantId/memory`. It - replaces `createMemory({ app, callerResolver })` and `registerMemoryRoutes`, - and `RouteDeps` no longer carries `grants`. `createMemory` only builds the - plane. -- **Breaking:** the package root exports only the public API. Internal - services and helpers (transform, retention, feed, share materialization, - corroboration, embed model registry, degrade metrics, FTS helpers), the test - fakes, and `resolveGrantConfig` are no longer exported. The distiller stays - at `@corbits/memory/distiller` and migrations at - `@corbits/memory/migrations`. -- **Breaking:** `runMemoryMigrations(config, { schema, ftsLanguage })` takes - the same `DBConfig` as Interchange `runMigrations` instead of a database - URL. `schema` names the host schema holding Interchange's `tenant` and - `principal` tables (the value passed to `runMigrations`, e.g. `"public"`); - memory's tables stay in the `memory` schema. `ftsLanguage` is required, and - the runner no longer reads `FTS_LANGUAGE` from the environment or accepts a - `log` option. -- Every migration file is idempotent and replayed on each run, with a 5 s - `lock_timeout`. The `memory._migrations` ledger is dropped. -- `MEMORY_GRANT_REQUIREMENTS` is read from `package.json` - `interchange.grantRequirements`, now the only declaration. - `MEMORY_CAPABILITY_IDS` is typed `string[]`. -- `prepack` runs `bun run build`. - -### Upgrading from 0.1.0 - -- Call the new `runMemoryMigrations` once; it upgrades a 0.1.0 database in - place with no data loss. -- After the upgrade, do not run 0.1.0 against the database: its ledger is - gone, so there is no downgrade. - -## [0.1.0] — 2026-09-25 - -First release on npm. - -[0.2.0]: https://github.com/corbitsdev/corbits-memory/releases/tag/v0.2.0 -[0.1.0]: https://www.npmjs.com/package/@corbits/memory/v/0.1.0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ebf7ec0..3f16695 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,68 +1,38 @@ # Contributing -Thanks for considering a contribution to Corbits Memory. +## Development -## Running it locally - -```bash -git clone https://github.com/corbitsdev/corbits-memory.git -cd corbits-memory -docker compose up -d # pgvector Postgres on localhost:5434 -cp .env.example .env # edit as needed — see README.md's Reference +```sh bun install -bun run db:setup # applies migrations/*.sql, idempotent -bun run build # compiles src/ to dist/, which the package publishes +bun run check ``` -Requires Bun 1.2+. `compose.yml` also runs a local Ollama for embeddings -(`docker compose exec ollama ollama pull nomic-embed-text`). Unit tests use -the in-repo `createFakeDocumentStore`/`createFakeSourceProvider` and need no -Postgres. See `IMPLEMENTATION.md` for how the pieces fit together. - -## Running the tests - -```bash -bun run typecheck && bun run test && bun run test:e2e -``` +`bun run check` runs typecheck, lint, format check and unit tests. `bun run format` rewrites the tree. -- `bun run test` runs the unit suite in `src/`; `bun run test:e2e` runs the - end-to-end suite in `e2e/`. The end-to-end tests drive the mounted routes and migrations - against a real pgvector Postgres: set `TEST_DATABASE_URL` to a server the - tests can create and drop databases on (for `docker compose up -d`, - `postgres://memory:memory-dev-password@localhost:5434/memory`). Each suite - creates its own database and drops it afterwards. Without - `TEST_DATABASE_URL` those suites skip. -- `bun run test:coverage` runs the unit suite with lcov + text coverage - reports. +Contributors sign the [CLA](CLA.md) on their first PR; the CLA bot explains how. -`bun run typecheck` (`tsc --noEmit`) must be clean before any commit. +`bun run test:e2e` drives the mounted routes and migrations against a real pgvector Postgres. `docker compose up -d` starts one on `localhost:5434` (plus Ollama for embeddings and a TEI reranker for manual runs), then run `TEST_DATABASE_URL=postgres://memory:memory-dev-password@localhost:5434/memory bun run test:e2e`. Each suite creates and drops its own database, and the suites skip when `TEST_DATABASE_URL` is unset. `cp .env.example .env` and `bun run db:setup` apply the schema for manual runs. `bun run test:coverage` reports lcov and text coverage over `src/` and `e2e/`. ## Migrations -`runMemoryMigrations` replays every file in `migrations/` on each run, so -every file must be idempotent. Never change a shipped file's effect on an -existing database: a changed constraint or a new column goes in a new -numbered file, because a guarded `ADD CONSTRAINT` keeps the old definition. - -## Branch and PR conventions - -- Branch off `main`; open PRs against `main`. -- Keep PRs scoped to one logical change — a mix of an unrelated refactor and a - feature makes review slower, not faster. -- Describe _why_ the change is needed in the PR description, not just what - changed; link any relevant issue. -- Make sure `bun run typecheck && bun run test` pass before requesting review. - -## Contributor License Agreement - -Contributions require agreeing to the project's CLA — see `CLA.md`. The CLA -bot will comment on your first PR with instructions if you haven't signed -yet. +`runMemoryMigrations` replays every file in `migrations/` on each run, so every file must be idempotent. Never change a shipped file's effect on an existing database: a changed constraint or a new column goes in a new numbered file, because a guarded `ADD CONSTRAINT` keeps the old definition. ## Commit messages Commit subjects and PR titles follow [Conventional Commits](https://www.conventionalcommits.org): `feat`, `fix`, `refactor`, `test`, `docs`, `build`, `ci`, `perf`, and `chore(release): x.y.z` for releases. Add `!` only for public API breaks: removed or renamed exports, changed signatures, newly required params. Peer and dependency range changes are `build(deps):` with no `!`. Keep subjects imperative, lowercase after the colon, 72 characters or less, and free of ticket IDs. -No ticket IDs in code or comments either. One logical change per commit where practical; describe the change, not the task that produced it. Every PR links its issue with a `Closes ` line in the PR body. + +## Releasing + +Releases are manual. On a clean, up-to-date `main`: + +```sh +npm version -m "chore(release): %s" +git push --follow-tags +gh release create "v$(node -p 'require("./package.json").version')" --generate-notes +npm publish +``` + +Bump minor only for breaking API changes; everything else is a patch. `prepack` builds `dist/` from the tagged commit. diff --git a/IMPLEMENTATION.md b/IMPLEMENTATION.md deleted file mode 100644 index cd6d3ab..0000000 --- a/IMPLEMENTATION.md +++ /dev/null @@ -1,811 +0,0 @@ -# Corbits Memory — Implementation Reference - -This is the detailed implementation reference: concrete files, tables, functions, -and wire shapes. For the "why standalone" / boundaries story, read -`ARCHITECTURE.md` first — this doc complements it, it does not repeat it. - -## Repo layout - -``` -src/ - index.ts # createMemory / createMemoryRoutes / mountWorkflowMemory - - mount-config.ts # MemoryConfig + loadMemoryConfig() — the mount config - config.ts # EngineConfig — the core vector-plane config (db + embed + rerank) - memory.ts # createMemory — add/search/list against store or pgvector - grant-tags.ts # resolveAccessTags + canAccessDocument (host grants) - workflow-mount.ts # mountWorkflowMemory — run-scoped /api/workflow-memory/* - sidecar-bundle.ts # @corbits/memory/sidecar-bundle factory (no client, no token) - tools.ts # MEMORY_TOOL_DEFINITIONS — sidecar binds names to run-scoped routes - http-client.ts # host-side HTTP client for tenant routes (imperative distill tick) - - log.ts # getLogger(["memory"]) from @intx/log - migrations.ts # runMemoryMigrations(dbConfig, { schema, ftsLanguage }) - ports/ # DocumentStore / SourceProvider + fakes - routes/ # the mounted tenant routes - mount.ts # createMemoryRoutes (HTTP sub-app) - - deps.ts # RouteDeps, caller(c) (context identity), grantGuard - add.ts, search.ts, list.ts, feed.ts - distiller/ # Resident distiller (CL-5869) — workflow + tick helpers - index.ts # createResidentDistiller, runDistillTick, buildDistilledClaim - workflow.ts # defineWorkflow + defineAgent with memory tools - tick.ts # imperative distill tick (host injects distill()) - claim.ts # pure claim body / gate / cursor helpers - db/ - schema.ts # Drizzle table defs (memory.* schema) - client.ts # createDb(config) -> { db (drizzle), sql (raw postgres-js) } - services/ - capture.ts # captureDocument, deriveFromRawCapture — the write path - search.ts # hybridSearch and every retrieval-candidate query - timeline.ts # listTimelineEvents — durable recent docs + grant-tag filter - transform.ts # transform_config CRUD + runTransform (replay) - feed.ts # capture feed cursor pull (CL-5868) - retention.ts # deprecate / tombstone / sweep ephemeral (CL-5871) - share-grants.ts # peer grant materialization on share (CL-5873) - core/ # framework-agnostic (chunking, embed/rerank, merge, schemas) -# DocumentStore adapters / tools live as sibling packages (not in this tree): -# @corbits/mem0-memory-adapter → github.com/corbitsdev/corbits-mem0-memory-adapter -# @corbits/supermemory-memory-adapter → github.com/corbitsdev/corbits-supermemory-memory-adapter -# @corbits/linear-tools → github.com/corbitsdev/corbits-linear-tools -migrations/ # pgvector schema, applied in filename order by scripts/db-setup.ts -scripts/db-setup.ts # runs the idempotent migrations against DATABASE_URL -compose.yml # pgvector + Ollama + reranker for local dev -``` - -The SDK has no server and no process entrypoint. `createMemory` takes - -the host's `Hono` app plus `{ config, grants? }` and mounts the -routes; each reads identity from the context (`caller(c)`) and guards via -`grantGuard`. Services take `{ db, sql, config }` explicitly (no module-level -singletons except the logger). Nothing has import-time side effects, so unit -tests exercise the routes and services directly without a listening server. - -Mounting does **not** verify the FTS language against the database at boot — -see the `memory_chunk` section below. A host is expected to either run -`runMemoryMigrations` itself (which verifies) or wire its own readiness -probe to call `verifyFtsLanguage`; without one of those, a language mismatch -surfaces as a runtime failure on the plane's first query, not at mount time. - -## Config - -There are two config types, both in the SDK: - -- **`EngineConfig`** (`src/config.ts`) — the core vector-plane config the DB - client and capture/search/transform services consume: `databaseUrl`, - `dbPoolMax`, `embed`, `rerank`. -- **`MemoryConfig`** (`src/mount-config.ts`) — what `createMemory` - - takes: just `{ memory: EngineConfig }`. `loadMemoryConfig()` builds one - from the environment; hosts may also construct it programmatically. Auth, - tenancy, and grants are the host's — none of that is config here. - -`loadMemoryConfig()` uses the same fail-loud helpers: `requireEnv(name)` -throws if unset/empty, `optionalEnv(name)` returns `undefined`, `intEnv(name, -fallback)` parses a positive integer or throws. - -| Var | Required? | Default | Notes | -| ----------------- | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `DATABASE_URL` | **yes** | — | the engine's own pgvector Postgres | -| `DB_POOL_MAX` | no | `8` | postgres-js pool size | -| `FTS_LANGUAGE` | no | `english` | text search config for the lexical channel; fixed into the generated column at migration time — changing it later requires rebuilding the column (recipe below), and `runMemoryMigrations` fails loudly if config and column disagree. Unqualified `pg_catalog` config names only — a schema-qualified config (`myschema.mycfg`) is rejected explicitly, both when configuring and when read back from an already-migrated column. | -| `EMBED_BASE_URL` | no | — | embed endpoint root, no path suffix; absent (with `EMBED_MODEL` also absent) => lexical-only, see below | -| `EMBED_MODEL` | no | — | model id/name passed to the embed endpoint; must be set together with `EMBED_BASE_URL` (both or neither — one without the other throws) | -| `EMBED_API_STYLE` | no | `"openai"` | `"openai" \| "tei" \| "ollama"` | -| `EMBED_API_KEY` | no | `undefined` | forwarded as `Authorization: Bearer ` | -| `RERANK_BASE_URL` | no | `undefined` | absent => search degrades to fusion-only | -| `RERANK_MODEL` | no | `undefined` | defaults to `bge-reranker-v2-m3` in the client | -| `RERANK_API_KEY` | no | `undefined` | forwarded as Bearer token to the rerank endpoint | - -**Lexical-only mode (CL-6287).** `EngineConfig.embed` is optional — leave both -`EMBED_BASE_URL`/`EMBED_MODEL` unset and the engine still constructs and -serves `add` + lexical `search` against a pgvector Postgres with no -embed endpoint configured. Dense retrieval is skipped rather than -attempted (no doomed HTTP call on every query), `add` still captures -documents (chunks stored, no vectors), and both verbs report a `degraded` -reason array — never a bare boolean, so a host can write one "is this -response degraded" check across both: `search` reports -`degraded: ["dense_unavailable", "lexical_only"]`; `add` reports -`degraded: ["embed_unavailable", "lexical_only"]` (or `["embed_unavailable"]` -alone when the endpoint IS configured but a specific embed pass failed — a -client error, timeout, or rejected chunk). The embed-model registry -(`ensureEmbedModel`/`activateEmbedModel`) is never reached in this mode. - -**Discoverability.** A host does not have to run a search to learn recall is -limited: `memory.capabilities.embeddingsConfigured` (on the `Memory` handle -`createMemory` returns) is `false` for a lexical-only engine, `true` -otherwise — known at construction, no query needed. A custom `documentStore` -that doesn't report its own `capabilities` defaults to `true` (this SDK -cannot introspect a vendor store it doesn't own); see -`DocumentStoreCapabilities` (ports/types.ts) for how a vendor store opts in. - -The replay/backfill pipeline (`runTransform`, `promoteGeneration` in -`services/transform.ts`) still requires an embed endpoint — re-deriving a -corpus is inherently a re-embedding operation — and fails loudly if run -against an engine with none configured; re-embedding documents captured while -lexical-only, once an endpoint is later added, is an open follow-up (not -implemented). - -The engine's `EngineConfig.rerank` carries -no `apiStyle` field of its own; `search.ts`'s `toRerankClientConfig` hardcodes -`apiStyle: "tei"` when building the client config, i.e. the engine currently -only wires a TEI-compatible reranker via env (Cohere/Voyage rerank backends -are reachable only through a per-`transform_config` `rerank.apiStyle`, not -through top-level env). - -**Model endpoints are trusted URLs — no SSRF guard, no self-host flag.** Every -embed/rerank endpoint the engine calls — its own `EngineConfig.embed`/ -`EngineConfig.rerank` (the capture embed pass, the dense-search query embed, -the embed-model-registry probe/activation, and `toRerankClientConfig`) and a -`transform_config`'s replay embed override (`buildEmbedClientConfig` in -`services/transform.ts`) — is treated as a trusted URL, exactly like -`DATABASE_URL`. There is no private-IP / SSRF filtering and no `allowSelfHost` -knob anywhere: a self-hosted endpoint on `localhost` or a private IP is just a -URL, indistinguishable from a managed provider. Model/replay endpoints are -configured by the operator (env) or named by a trusted caller in a -`transform_config`, never by an unauthenticated request — so operators who need -egress control front the endpoints with an allowlisting proxy. - -## Data model (`src/db/schema.ts` + `migrations/*.sql`) - -All tables are Drizzle-defined in `db/schema.ts`, DDL'd in `migrations/` -(applied by `runMemoryMigrations`; every file is idempotent and replayed on -each run, so there is no ledger). No memory table has a foreign key into any -control-plane table — `tenant_id`/`principal_id`/source refs are plain `text`. - -### `memory_document` - -The stable logical row for a captured source. Unique on -`(tenant_id, adapter, external_ref)` — this triple is the dedupe/identity key -every capture upserts against. Document access is **grant tags**: -`access_tags text[]` (resource strings in grant-pattern space; see -`docs/AUTHZ-DOCUMENT-ACCESS.md`). `attributes` is a flat jsonb bag of scalars. -`last_seen_at` bumps on every re-capture, even a content-hash NOOP. - -### `memory_version` - -The versioned body of a document. `version` is a monotonic integer scoped to -`(document_id, generation)` — **not** globally per-document — per the -`memory_version_document_generation_version_uniq` unique index (baseline -schema). `status` tracks `'active'|'superseded'|'deprecated'|'archived'|'tombstoned'`; -only one `active` row exists per `(document_id, generation)` at a time (the -capture path enforces this by flipping the prior active row to `superseded` -before inserting a new one). `content_hash` is the NOOP-check key. Attribution -columns: `created_by_principal_id`, `created_by_kind` -(`'human'|'agent'|'system'|'adapter'`), `generator_agent_id`. Authority -columns (`authority`, `actor_count`, `has_social_signal`, `source_class`) are -a **snapshot computed once at capture time** (`computeAuthority`, never -recomputed retroactively). `raw_capture_id` points at the immutable source row -this version was derived from. `generation` (default `'live'`) is the -replay-generation tag: the normal add path always writes -`'live'`; a replay (`runTransform`) writes its own `transform_run.id` instead, -so a replayed corpus's versions never collide with, or even become visible -alongside, the live ones unless a caller explicitly searches that generation. - -### `memory_chunk` - -An ordered slice of a version's text, keyed by `(version_id, ordinal)` -(unique). Carries a generated-always `text_fts tsvector` column (GIN-indexed) -that powers the lexical search channel — this is the only place FTS is -computed; no separate FTS table exists. Its language comes from -`FTS_LANGUAGE` at migration time; the query side binds the same configured -language as a `regconfig` parameter. The invariant is verified twice, both -read-only against the catalog: `runMemoryMigrations` checks after -applying (the deploy step), and the memory plane runs the same check -once, memoized, before its first query (the serving path) — so a mismatch -or unmigrated schema fails loudly on first use regardless of who ran the -migrations. **The serving-path check only runs when something actually -calls it** — `search()`/`capture()` invoke it lazily and memoize the result, -but nothing forces that first call to happen at boot. A host that mounts the -engine without running `runMemoryMigrations` itself and without wiring a -readiness probe will not learn about a language mismatch until the first -real query or capture fails — not at startup. Hosts that want a boot-time -guarantee **must** call the exported `verifyFtsLanguage` from their own -readiness probe; it is not optional belt-and-suspenders, it is the only way -to get a boot-time check if this SDK instance isn't the one that migrated. -Chunks are **never** reused across versions — every new version gets a -fresh full insert of its own chunks. - -**Changing `FTS_LANGUAGE` on an already-migrated database** (the mismatch -`verifyFtsLanguage` throws on) requires rebuilding the generated column — -`runMemoryMigrations` adds the column only if it is missing and will not -retroactively alter an existing one. One-time recipe (verified against a live -`postgres:16` instance): - -```sql -BEGIN; -DROP INDEX IF EXISTS memory_chunk_text_fts_idx; -ALTER TABLE memory_chunk DROP COLUMN text_fts; -ALTER TABLE memory_chunk ADD COLUMN text_fts tsvector - GENERATED ALWAYS AS (to_tsvector('', "text")) STORED; -COMMIT; - --- Separate statement/connection — CREATE INDEX CONCURRENTLY is rejected --- inside any transaction block, unconditionally, since Postgres 8.2. It --- cannot be combined with the BEGIN/COMMIT block above. -CREATE INDEX CONCURRENTLY memory_chunk_text_fts_idx ON memory_chunk USING gin (text_fts); -``` - -Both `ALTER TABLE` statements take an `ACCESS EXCLUSIVE` lock and force a -full table rewrite (dropping then re-adding a `STORED` generated column -always rewrites) — plan for a stall on `memory_chunk` for the duration on -a populated database; run in a maintenance window. Only unqualified -`pg_catalog` config names are supported; a schema-qualified config on this -column is rejected explicitly by `verifyFtsLanguage` (with this same recipe -in the error) rather than silently mis-parsed. - -### `memory_entity` / `memory_edge` - -Lightweight graph rows. `memory_entity` has no unique constraint; dedupe on -`(tenant_id, kind, identifiers)` is done in application code -(`upsertEntity` in `capture.ts`, an exact-match linear scan per kind). Same for -`memory_edge` (dedupe on the full `(tenant_id, rel, from, to)` tuple, -`upsertEdge`). `rel` is constrained (DB CHECK + arktype, single source of -truth in `src/core/enums.ts`) to -`mentions|about|authored_by|involves|part_of|derived_from|supports|contradicts|supersedes`; -`from_type`/`to_type` stored in the DB are -`document|version|chunk|entity`. Adapter-facing edge hints may also use -`native` as a planning-time principal ref; capture resolves it to an -`entity` row (`kind=principal`) before insert. A lockstep test asserts the -migration CHECK sets match the TS constants. - -### Provenance and lineage (claim-bearing substrate) - -Two axes on `memory.version`, orthogonal to ranking priors: - -| Column / field | Values | Meaning | -| ------------------------ | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -| `provenance` | `stated` \| `inferred` \| `unknown` | How content was obtained. Capture defaults to `stated`; distilled claims write `inferred`. Existing rows default `unknown`. | -| `source_class` (lineage) | `native` \| `imported` \| `derived` | Data lineage. Adapters write `native` (or `imported` for bulk import); distilled claims write `derived` via `AdaptedDocument.lineageClass`. | - -Ranking priors (`AdaptedDocument.sourceClass`: `native|thread|channel|call|record`) feed `computeAuthority` only and are **not** written to the version `source_class` column — that was a latent CHECK violation fixed when the axes were split. - -A derived claim is a normal version with `provenance: inferred`, `lineageClass: derived`, and a `derived_from` edge to the source version (or document). Core never runs inference; it only accepts the shape. - -### Temporal model - -See `docs/TEMPORAL.md`. On `memory.version`: - -| Column | Role | -| ---------------------------- | ------------------------------------------------------------ | -| `occurred_at` | Effective time the content refers to | -| `ingested_at` | When the plane learned it (no separate `asserted_at`) | -| `temporal_class` | `event` \| `deadline` \| `state` \| `lesson` — ranking prior | -| `valid_from` / `valid_until` | Optional validity window | - -Search multiplies fused scores by `temporalRecencyMultiplier` (class-aware). Timeline and search both filter `generation` (default live) so replay rows never leak into live views. - -### `memory_embed_model` - -Per-tenant registry of embed models and their discovered dimensionality -(`discoverModelDims`/`probeEmbedDims` — dims are **never** hard-coded, always -probed live against the endpoint). Unique on `(tenant_id, model_key)`, where -`model_key` is `sha256(baseUrl|modelId).slice(0,16)` (`computeModelKey`). - -Two write paths (CL-5872): - -- **`ensureEmbedModel`** — upserts the registry row with `status = 'ready'`, - creates the per-model table + indexes. **Never** flips the tenant's active - dense table. Used by staged `runTransform` when the config's embed model - differs from live. -- **`activateEmbedModel`** — `ensure` then `UPDATE … SET status = 'active'`. - Used only by the **live capture** path and by **`promoteGeneration`** when - cutover should make the generation's embed model serve live dense search. - -"Active" for live dense search is the most-recently-`updated_at` row with -`status = 'active'` (`resolveActiveEmbedTable`). Replay dense search uses -`resolveEmbedTableByModelKey` against the owning transform config's embed -`model_key` — ready or active — so staged generations never steal live. - -Optional `archived_live_generation` and `archived_live_model_key` on -`transform_run` record which generation and dense model held live rows before -promote (for demote/rollback of both corpus and dense search). - -### Dynamic per-model vector tables: `memory_embedding_` - -Not in `db/schema.ts` (no fixed shape — dimensionality varies by model) and -not in any migration file. Created at runtime by -`activateEmbedModel` (`embed-model-registry.ts`) the first time a given -`(baseUrl, modelId)` pair is used: - -```sql -CREATE TABLE IF NOT EXISTS memory_embedding_ ( - chunk_id text PRIMARY KEY, - tenant_id text NOT NULL, - embedding vector(), - CONSTRAINT memory_embedding__chunk_fk - FOREIGN KEY (chunk_id) REFERENCES memory_chunk (id) ON DELETE CASCADE -) -``` - -plus a `(tenant_id, chunk_id)` index (created on every activation, so it -retrofits onto older tables). The FK completes the hard-delete cascade -chain document -> version -> chunk -> embedding. `CREATE TABLE IF NOT -EXISTS` cannot add the FK to a table created before it existed; that gap is -accepted (no populated pre-FK installs exist). If hard deletes are ever -introduced against such a table, run once, per embedding table: - -```sql -ALTER TABLE memory_embedding_ - ADD CONSTRAINT memory_embedding__chunk_fk - FOREIGN KEY (chunk_id) REFERENCES memory_chunk (id) ON DELETE CASCADE; -``` - -plus an HNSW index: `vector_cosine_ops` up to 2000 dims (falling back to -`ivfflat` if the Postgres/pgvector build lacks the `hnsw` access method), or -a `halfvec` expression index (`halfvec_cosine_ops`, pgvector >= 0.7) for -2001–4000 dims — pgvector's `vector`-typed indexes cap at 2000 dims. Models -above 4000 dims are rejected at activation (`MAX_EMBED_DIMS`) since no index -type could serve them. `MAX_EMBED_DIMS` is now pinned to -`HALFVEC_INDEX_MAX_DIMS` (4000, pgvector's halfvec index cap), narrowed down -from the previous 4096 — a model in the 4001–4096 range that used to -activate and only fail later (no index type covering it) now fails loudly -at activation instead. The dense query's `ORDER BY` is generated by -`cosineDistanceExpr` from the same module so it always matches the indexed -expression. The table name is -validated against `EMBED_TABLE_NAME_PATTERN` -(`/^"memory"\."embedding_[a-f0-9]{16}"$/`) both when computed and again every -time it's read back from `memory_embed_model`, before ever being -string-interpolated into raw SQL — this is the only place in the codebase a -computed identifier is spliced into DDL/DML. - -### `raw_capture` - -The immutable, append-only substrate (the raw-capture layer). Stores the exact add/ingest -request payload (`adapter`, `occurred_at`, `document`) as JSON in `raw_text` -(there's also a `raw_bytes bytea` column for non-textual payloads, currently -unused by any write path — everything captured today is JSON). Deduped on -`(tenant_id, source_hash)`, where `source_hash` is -`sha256(stableStringify({adapter, occurredAt, document}))` — a byte-identical -recapture reuses the existing row (`insertOrReuseRawCapture`) rather than -inserting a duplicate; the table never has rows updated or deleted by -ingestion. - -### `transform_config` - -A named, versioned recipe of derivation + retrieval-tuning knobs -(`TransformConfigParams`: `chunk` — only `token.recursive` is a valid -strategy today — `embed`, optional `rerank`, `authorityWeight`, -`recencyHalfLifeDays`, `mmrLambda`, `overfetch`). Unique on -`(tenant_id, name, version)`; re-`POST`ing the same `name` mints -`version = max(existing) + 1` rather than colliding (`createTransformConfig`). - -### `transform_run` - -One execution of a `transform_config` against a (possibly scoped) slice of -`raw_capture`. `generation` is this run's own id, unique -(`transform_run_generation_uniq`) — so a generation always resolves back to -exactly one run and therefore one config at search time -(`resolveGenerationSearchParams`). `status` is `'running'|'completed'|'failed'`; -`scope` is jsonb (`{ adapter?, since?, until? }`, filtering on -`raw_capture.fetched_at`). `runTransform` never throws for a mid-run -derivation failure — it marks the run row `'failed'` with `error` recorded and -returns the run summary either way. - -## Capture flow (`services/capture.ts` — `captureDocument`) - -1. **`adaptAndPlan(document)`** (`core/adapt-and-plan.ts`, pure, no I/O): - validates `title`/`kind` are non-empty, re-chunks every incoming - `AdaptedDocumentChunk` through the chunker port (default - `chunkTokenRecursive`, caps `DEFAULT_CHUNK_CAPS` = 700 max / 40 min / 60 - overlap tokens — `adaptAndPlan`'s internal `rechunk` always applies these - defaults regardless of chunker, which is why the live path is unaffected - by a replay's custom caps), and computes `contentHash` over - `title + kind + externalRef + stableStringify(attributes) + joined chunk -text` (`core/hash.ts`) — this hash is the NOOP-check key. -2. **`insertOrReuseRawCapture`**: hashes the raw wire payload - (`computeSourceHash`, independent of `contentHash` — this one covers the - _unadapted_ input including `adapter`/`occurredAt`) and looks it up by - `(tenantId, sourceHash)`; reuses the existing `raw_capture` row or inserts - a new one, all inside the same transaction as the derived rows. -3. **`deriveVersionInTransaction`** — the single derivation core shared by - live capture and replay: - - No existing `memory_document` for `(tenantId, adapter, externalRef)` - → insert a new document row + a version at `version = 1`, - `supersedesVersionId = null`. - - Existing document, and its current `active` version (scoped to this - `generation`) has the **same** `contentHash` → **NOOP**: bump - `last_seen_at` only, write nothing else, return - `{ status: "noop", chunks: 0 }`. - - Existing document, content changed → flip the prior active version to - `status: "superseded"`, insert a new version at `version + 1` with - `supersedesVersionId` pointing at it, update the document's mutable - fields (`title`, `access_tags`, `attributes`, `last_seen_at`). -4. **`insertChunksAndGraph`**: inserts every plan chunk fresh (chunks are - never reused across versions), then best-effort upserts entity hints - (`upsertEntity`) and edge hints (`upsertEdge`) — these are independent of - version and never rolled back if a later step fails within the same - transaction (they're just additional statements inside it). -5. **After the transaction commits** — `embedInsertedChunksWithConfig`: - resolves/activates the tenant's embed model (`activateEmbedModel`, probing - dims live and creating the per-model vector table if needed), then - `embedChunks` embeds and inserts vectors for the freshly-inserted chunks. - This step is **best-effort**: an embed-client failure (timeout, HTTP - error) or a per-chunk dims mismatch is logged via - `log.warn` and never thrown — the chunk rows are already durable in - Postgres, and the module's own comments note a later re-embed pass could - pick up anything left unembedded (no such background pass exists yet in - this repo; chunks left unembedded simply never populate the dense - channel for the query — they're still found by lexical/FTS). Any of these - failure modes sets `degraded: true` on the `CaptureResult`, surfaced by - `POST /api/tenants/:tenantId/memory/add` as a `degraded` field in its response — the add - still succeeded (chunks are durable and lexically searchable), only the - dense/vector channel for those chunks is incomplete. - -**Request-size guards independent of embedding.** `index.ts` caps the whole -request body at `MAX_REQUEST_BODY_BYTES = 10 MB` (`hono/body-limit`, -rejecting oversized bodies with `413` before they're even parsed as JSON); -`core/schemas/adapted-document.ts` separately caps a single chunk's text at -`MAX_CHUNK_TEXT_CHARS = 100,000` chars, the number of chunks per document at -`MAX_CHUNKS_PER_DOCUMENT = 2,000`, `title` at `MAX_TITLE_CHARS = 500`, and -`kind` at `MAX_KIND_CHARS = 200` — a payload well under the body-size limit -could otherwise still carry pathologically many or large chunks. - -The live path (`captureInTransaction` → `deriveVersionInTransaction`) always -writes `generation = LIVE_GENERATION` (`"live"`, `core/generation.ts`) and -always resolves/reuses its own `raw_capture` row. `deriveFromRawCapture` is -the same core function called directly by a replay with an **existing** -`raw_capture_id` and the run's own generation — a replay never writes a new -`raw_capture` row (the raw-capture corpus is read-only from that path). - -## Search pipeline (`services/search.ts` — `hybridSearch`) - -Entry point, one query in, one ranked/citable hit list out. `k` is clamped to -`[1, MAX_K=100]`, default `DEFAULT_HYBRID_TOP_K = 8`. An empty `query` string -is only accepted if `kinds` or `entityIds` is provided (structured-filter-only -search); otherwise it throws `MemorySearchInputError` (400). - -**Living relevancy (CL-5867):** after fusion, `attachCorroborationCounts` loads -`supports`/`contradicts` edge counts per version. Ranking multiplies by -`corroborationFactor` (bounded [0.7, 1.3]); evidence:strong also requires the -gate in `core/corroboration.ts` (stated human **or** support count ≥ floor). -Capture-time `authority` is never rewritten. See `docs/RELEVANCY.md`. - -**Wire attribution (CL-5870):** `attachDerivedFrom` loads `derived_from` edges -onto candidates; `toHit` emits provenance, source/temporal class, occurred_at, -valid_until, corroboration counts, and derived_from. The plane maps these into -additive `SearchItem.attribution` (and `DocumentStoreSearchItem.attribution`). -**List/timeline** stays document-title oriented and does not attach the full -attribution block (version-level fields are search/feed concerns). - -**Retention (CL-5871):** `memory.version.retention_class` + migration -`0007_retention.sql`. Plane helpers in `services/retention.ts` -(`deprecateVersion`, `tombstoneDocument`, `hardDeleteDocument`, -`sweepEphemeral`, `setRetentionClass`). Search accepts `includeDeprecated` -so lexical/dense can include `status IN ('active','deprecated')`. See -`docs/RETENTION.md`. - -**Capture feed (CL-5868):** `memory.feed({ after, limit, excludeGenerator? })` -and `GET .../memory/feed` pull live versions ordered by `feed_seq` (migration -`0006_capture_feed.sql`). Grant-checked like search. See `docs/FEED.md`. - -1. **Generation resolution** — `generation` defaults to `LIVE_GENERATION`. - For any non-live generation, `resolveGenerationSearchParams` (transform.ts) - looks up the owning `transform_run` → `transform_config` **for the search - tenant** and pulls its tuning knobs (`authorityWeight`, - `recencyHalfLifeDays`, `mmrLambda`, `overfetch`, `rerank`); every field it - doesn't supply falls back to the engine's own defaults. Live search never - pays for this lookup. Cross-tenant generation ids resolve to `null` - (engine defaults) so embed overrides (including `apiKey`) cannot leak. -2. **Lexical channel** — `fetchLexicalCandidates`: Postgres full-text search - (`ts_rank` against `plainto_tsquery` in the configured `FTS_LANGUAGE`, - bound as a `regconfig` parameter, over - `memory_chunk.text_fts`), joined to `memory_version` (filtered to - `status = 'active'` and the resolved `generation`) and `memory_document` - (tenant-scoped only — document access is grant-tag post-filter in the plane), - optionally further filtered by - `kinds` and/or `entityIds` (via a sub-select against `memory_edge`). - Overfetches up to `overfetchLimit` rows, non-deduped, per-chunk. -3. **Dense channel** — `fetchDenseCandidates`: embeds the query - (`embedTexts`), resolves the dense table: - - live generation → `resolveActiveEmbedTable` (tenant active model) - - staged generation → `resolveEmbedTableByModelKey` for the transform - config's embed model (ready or active; never activates) - runs a raw-SQL cosine-distance ANN query via `cosineDistanceExpr` - (`e.embedding <=> $vector` up to 2000 dims, or the matching - `(e.embedding::halfvec(N)) <=> $vector::halfvec(N)` expression above that - so the halfvec HNSW index is used) - against that table joined back to - `memory_chunk`/`memory_version`/`memory_document` with the - **same tenant-only scope** as the lexical channel (no mini-ACL in SQL). - Returns `null` (not an error) when there's no active embed - model yet or the query is empty; a thrown error from the embed call or - the SQL itself is caught by the caller and also folds into `null`/degraded - — the dense channel never fails the whole search. -4. **RRF fusion** — `fuseRrf` (`core/hybrid-search.ts`): combines the - lexical and dense per-channel rank orders (never raw scores — they're on - incomparable scales) via Reciprocal Rank Fusion, - `score = Σ 1/(60 + rank)`. -5. **Per-document dedupe** (non-reranked path only) — - `dedupeCandidatesPerDocument`: collapses to the single highest-scoring - chunk per `documentId`, using `authorityWeightedScore` (`relevance * (1 + -0.5 * authority)`) as the rank prior, tie-broken by recency. -6. **Rerank** (only if a rerank endpoint is configured — engine env - `RERANK_BASE_URL`, or the replay generation's `transform_config.params.rerank`): - dedupe (without the authority prior — authority is applied later on this - path, never twice), take the top `RERANK_CANDIDATE_LIMIT = 50`, call - `rerankDocuments` (cross-encoder), sort by rerank score. -7. **Bounded authority/recency boosts** — `applyBoosts`: normalizes the - active-stage score (rerank score, or fused RRF score on the degraded path) - to `[0,1]` within the batch, then multiplies by - `authorityBoostMultiplier` and `recencyBoostMultiplier`, both clamped to - `[0.7, 1.3]` — a boost can never let a weak match outrank a strong one on - its own. Take the top `MMR_POOL_SIZE = 20`. -8. **MMR diversity pass** — `mmrRerank` (`core/mmr.ts`): greedy pick - maximizing `relevance - λ * maxSimilarityToAlreadyPicked` (`λ = 0.7` - default, or the replay config's `mmrLambda`), using vectors pulled fresh - from the active embedding table (`fetchChunkVectors`). Items without a - vector are never dropped — appended by score after every vector-bearing - item is placed. Produces the final top-`k` order. -9. **Degrade path** — if reranking fails (network error, non-2xx) or was - never configured, the pipeline falls back to - `dedupeCandidatesPerDocument(mergedRows, true, authorityWeight).slice(0, -k)` (fused + authority-weighted order, no MMR) and reports - `degraded: ["rerank_unavailable"]`. If dense retrieval failed/unconfigured, - `degraded` includes `"dense_unavailable"` and lexical alone answers. -10. **Finishing** — `attachEntityIds` joins in each surviving document's - entity edges; `toHit` builds the wire `SearchHit` (citation/open-target - resolution via `openTarget`, mapping known adapters — `artifact`, `task`, - `workflow_run`, `mail` — to a deep-linkable `{type, id}`, else a generic - `{type: "memory", id: documentId}`). -11. **Evidence** — `deriveHybridEvidence`: `"none"` if zero hits; `"weak"` if - the lexical channel contributed zero rows (a dense-only result never - reports `"strong"`); otherwise `deriveEvidence` on the lexical rows — - `"strong"` requires **both** the top-ranked hit's raw `ts_rank ≥ -STRONG_RANK_FLOOR (0.05)` **and** its authority `≥ -AUTHORITY_STRONG_FLOOR (0.3)`; else `"weak"`. - -Tenant isolation is unconditional and first in every query (`tenant_id` -filtered before grant-tag / creator document access); every table/channel is scoped that -way, with no exception. - -## Raw + replay (the replay pipeline — `services/transform.ts`) - -`raw_capture` is the immutable substrate every replay reads from and never -writes to. A `transform_config` is a named/versioned recipe -(`createTransformConfig`, `listTransformConfigs`) capturing chunk/embed/rerank - -- retrieval-tuning knobs. - -`runTransform(configId, scope?)`: - -1. Loads the config, mints a new `runId` = the run's own `generation`. -2. Inserts a `transform_run` row (`status: 'running'`). -3. Selects every `raw_capture` row in `scope` (adapter/since/until filters on - `fetched_at`; an empty scope = a full tenant backfill). -4. For each row: re-parses its stored JSON payload back into a `CaptureInput` - (`parseRawCapturePayload`, itself validated through - `RawCapturePayloadSchema` — re-hydrating from storage is treated as its - own trust boundary, never a blind `JSON.parse`), then calls - `deriveFromRawCapture` with the config's own chunker - (`chunkTokenRecursive` with the config's caps) and embed client config, - targeting the run's `generation` — never the live one. Embed tables are - **ensured** (`ensureEmbedModel`), never activated, so live dense search is - untouched until an explicit promote. -5. On completion, updates the run row: `status: 'completed'`, `rawCount`, - `versionCount`. On any exception mid-loop, catches it, logs it, and marks - the run `'failed'` with `error` set — `runTransform` itself never throws - to its caller; callers always get a run summary. -6. `resolveGenerationSearchParams` is how `hybridSearch` later maps a - generation back to its config's search-tuning knobs (authority weight, - recency half-life, MMR λ, overfetch, rerank config) **and** dense - `modelKey` for generation-scoped table resolution. - -**Promote / demote (staged cutover):** - -- `promoteGeneration` — requires `status = 'completed'`. Snapshots the - pre-promote active `model_key`, activates the staged embed model (sole - active for the tenant; peers demoted to `ready`), then swaps generation - tags (`live` → archive, staged → `live`). Records - `archived_live_generation` + `archived_live_model_key` for demote. If the - version swap fails after activate, re-activates the prior model_key (or - clears active when there was no prior). -- `demoteGeneration` — re-activates `archived_live_model_key` (fail-closed if - the registry row is gone), or clears all active models when no prior was - recorded, then restores archive → live and staged corpus back onto the run - generation. - -Plane methods (engine DocumentStore only): `createTransformConfig`, -`listTransformConfigs`, `runTransform`, `promoteGeneration`, -`demoteGeneration` on `Memory`. Custom/fake stores omit these methods. - -**Host privilege:** transform/promote/demote take `tenantId` (and config/run -ids) only — no principal, no HTTP routes. They rewrite live generation and the -active dense embed model for a tenant. The host must treat them as admin -operations (grant-gate before calling); never expose them unauthenticated to -end-user agents. - -## Mounted routes - -`createMemoryRoutes(deps)` serves these once the host mounts it at -`/api/tenants/:tenantId/memory`. Identity is the request - -principal read off the Interchange context (`caller(c)` → -`{ scopeId: principal.tenantId, subjectId: principal.id }`); clients never send -`tenant_id`/`principal_id` — the handlers only read title/text/query/limit/access_tags/share. -Each route is guarded with `grantGuard(deps, action)`, which applies the host's -`requireGrant("memory", action)` when provided (else a pass-through). - -**Machine callers (CL-6286):** `RouteDeps.callerResolver` (passed to -`createMemoryRoutes`) lets a host resolve identity for a caller -that never goes through its tenant-session middleware — e.g. a workflow-run -child authenticating with its own sidecar bearer token. Unset by default -(every existing host is unaffected). When set, `resolveCaller` (`deps.ts`) -runs ahead of `requirePrincipal`/`grantGuard`, calls the resolver, parses its -return with arktype (non-empty `tenantId`/`principalId` — a malformed -resolver return is a host bug and gets `500`, not `401`), and seats the -result as the context `principal`/`tenant` so the exact same -`requireGrant`/`authorize` path a browser caller gets applies to the machine -caller too. **Migrating a host off a hand-rolled parallel surface** (like a -`createWorkflowMemoryRoutes`-shaped workaround) onto `callerResolver`: that -kind of surface commonly also carries a per-run write-rate limiter and a -request payload cap that this package does not implement (see CL-6286's PR -body for why) — re-home both as host middleware before deleting the old -surface, or a migrating host silently loses them. - -| Method + path | Grant action | Request body | Response | -| ------------------------------------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `POST /api/tenants/:tenantId/memory/add` | `add` | `{ title, text, access_tags?, share? }` | `200 { documentId, versionId }`; `400` on validation | -| `POST /api/tenants/:tenantId/memory/search` | `search` | `{ query, limit?, kinds?, entity_ids?, sources?, includeEvidence? }` (limit 1–50; `kinds`/`entity_ids`/`sources` narrow retrieval before fusion; unset or `[]` = unfiltered; `includeEvidence` adds a short evidence string when true) | `200 { items[], evidence?, degraded? }`; `400` on bad input | -| `GET /api/tenants/:tenantId/memory/list` | `search` | query `?limit=` (1–100, string on the wire) | `200 { events: [{ at, title, source, tenantId, principalId }] }` — durable recent documents for the caller's scope, filtered with grant-tag access (`canAccessDocument`). One event per document (active live version). | -| `GET /api/tenants/:tenantId/memory/feed` | `search` | query `?after=&limit=&exclude_generator=` | `200 { entries[], nextCursor }` — cursor pull of new live versions. See `docs/FEED.md`. | -| `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` | `forget` | `{ reason? }` | `200 { documentId, versions }`; `403` unless caller is the document's creator; `404` unknown document. Tombstones — content is redacted, not archived; see docs/RETENTION.md. | -| `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` | `purge` | none | `200 { documentId, deleted, reason? }`; `403` unless caller is the document's creator; `404` unknown document. Hard-deletes the row — irreversible; refused while a `durable` version is untombstoned. | -| `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` | `forget` | `{ retention_class }` | `200 { versionId, documentId, status }`; `400` invalid class; `403` unless caller is the version's creator; `404` unknown version. | - -`createMemoryRoutes` serves these seven HTTP -routes (add, search, list, feed, forget, purge, retention-class). -**Capture** (`services/capture.ts`) is the write path inside `add`. -**Search** (`services/search.ts`) is hybrid retrieval. Deployed agents do -not call these tenant routes with a git-installed client; they use the -sidecar mount below. The plane surface is `add` / `search` / `list` / -`close`, plus optional transform methods when backed by the engine -DocumentStore (`createTransformConfig`, `listTransformConfigs`, -`runTransform`, `promoteGeneration`, `demoteGeneration`) and optional -retention methods (`tombstoneDocument`, `hardDeleteDocument`, -`setRetentionClass`, `sweepEphemeral`, `deprecateVersion`) — see -docs/RETENTION.md. Inference stays on the host. Host workers that already -have HTTP to the tenant tree can use `createMemoryHttpClient` -(`src/http-client.ts`) — that is a host client, not the agent sidecar. - -### Run-scoped sidecar (`mountWorkflowMemory`) - -`src/workflow-mount.ts`, re-exported from the barrel. Parallel to the -tenant tree — do not fold agent-bearer auth into `createMemoryRoutes`. -`package.json` exports `@corbits/memory/sidecar-bundle` → -`src/sidecar-bundle.ts`. - -```ts -import { mountWorkflowMemory } from "@corbits/memory"; - -mountWorkflowMemory(workflowMemoryApp, { - memory, - agentToken: { verify, resolveRun }, -}); -app.route("/api/workflow-memory", workflowMemoryApp); -``` - -| Constant | Value | -| --------------------------- | -------------------------------- | -| `WORKFLOW_MEMORY_BASE_PATH` | `/api/workflow-memory` | -| `HUB_CREDENTIAL_HANDLE` | `hub` | -| `SIDECAR_BUNDLE_ID` | `@corbits/memory/sidecar-bundle` | - -**Auth.** Every route sits behind middleware: `agentToken.verify(c)` reads -the presented `Authorization`; `agentToken.resolveRun` looks up -`x-workflow-run-address`. Same **401** body whether the bearer is -unrecognized, the address names no run, or the run's tenant is not the -token's tenant. No `requireGrant`. No tenant override. Scope on context: -`workflowRunScope` `{ tenantId, principalId, runId }`. - -**Wire (relative to the mount; sidecar never names a host).** Responses -are `{ data: … }` on success. Sidecar `requires`: `capabilities`, -`address` (run address). Tool results are JSON strings. - -| Tool name | Method + path | Notes | -| --------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| `memory_add` | `POST /add` | Body same as tenant add. Forces `share.tenant = true` (team share; explicit share only widens). Capture path is `memory.add` → `captureDocument`. | -| `memory_search` | `POST /search` | Body same as tenant search; `visibleTags` = `[tenantTag(tenantId)]`. | -| `memory_list` | `GET /list?limit=` | Same grant-tag visibility as search (`visibleTags` team tag). | -| `memory_feed` | `GET /feed?after=&limit=&exclude_generator=` | **501** if `memory.feed` is undefined on this plane. | - -Unknown tool name → tool error (`isError: true`). Non-HTTP hub credential -kind → error. `callMemoryRoute` sends `x-workflow-run-address` and -optional JSON body; mediated `credential.fetch` injects the bearer and -pins origin. - -`callerResolver` on the **tenant** routes is a different seam (machine -caller through the same grant path). Do not treat it as a replacement for -`mountWorkflowMemory`. - -### Share materialization (CL-5873) - -`share.principals` on `add` still mints owner tags, and when the host grant -store implements `WritableGrantStore.putGrant`: - -1. Appends `memory.doc:` to the document's `access_tags`. -2. Writes one allow/`search` grant per peer on that resource, origin - `system`, with `conditions.memoryShare` audit payload. -3. `createMemory` merges `MEMORY_SHARE_CONDITION_REGISTRY` so those - condition keys are not fail-closed-skipped by `@intx/authz`. - -Without a writable store: tags only + warn log (peers need host grants). -Audience widening uses `splitAudienceWiden` (write-narrow-then-widen) and -`shareWidenReceipt` on version attributes after source-owner approval. -Ask-on-read remains design-only (fail-closed). - -### Timeline wire fields (vs the old CaptureLog ring) - -The process-local CaptureLog hardcoded `source: "api"` and recorded the -HTTP caller's `subjectId` as `principalId` at capture time. The durable -timeline maps different columns: - -| Wire field | Source column / meaning | -| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `at` | `memory_document.last_seen_at` (ISO) — re-captures rise in the feed | -| `title` | `memory_document.title` | -| `source` | `memory_document.adapter` (HTTP add defaults to `"http"`, not `"api"`) | -| `tenantId` | `memory_document.tenant_id` | -| `principalId` | `memory_version.created_by_principal_id` of the active live version (empty string when null) — the capturing actor stored on the version, not the request principal of a later timeline read | - -### Document access (grant tags) - -Document access is Interchange authz — **not** a mini-ACL. - -- Write path: `resolveAccessTags` always writes `memory.owner:` and - merges optional `accessTags` / share sugar (`tenant`, peer `principals`, - explicit `tags`). Stored on `memory.document.access_tags`. -- Read path (search + list): `canAccessDocument` — creator always allowed; - otherwise `authorize(grantStore, principal, tenant, tag, "search")` for any - tag on the document. -- SQL retrieval is **tenant-scoped only**. Document access is grant-tag - post-filter in the plane (`canAccessDocument`); there is no SQL mini-ACL. -- HTTP `POST /add` accepts `access_tags` and/or `share` — not product `acl` - modes or block lists. - -See `docs/AUTHZ-DOCUMENT-ACCESS.md`. - -## Observability - -The SDK does no logging or error-reporting setup of its own — that belongs to -the host app. Routes log a one-line `console.error` on a 5xx-class failure and -return a generic error body; the host's middleware owns request logging and any -Sentry/OTel wiring. - -## Local dev - -```bash -docker compose up -d # pgvector + Ollama + reranker -docker compose exec ollama ollama pull nomic-embed-text -cp .env.example .env -bun install -bun run db:setup # apply the memory schema, idempotent -bun run test # unit suite (no external services) -``` - -`compose.yml` provisions the pgvector Postgres (`memory` db, host port -`5434`), an Ollama embeddings server (`:11434`), and a TEI reranker (`:8085`). -The engine **never embeds internally** — when `EMBED_BASE_URL` is set, it must -point at a real endpoint. A model endpoint is just a URL + capability options, -trusted the same as `DATABASE_URL`: - -- **Local default**: Ollama at `http://localhost:11434` - (`EMBED_API_STYLE=ollama`, `EMBED_MODEL=nomic-embed-text`). -- **Paid provider**: e.g. `EMBED_BASE_URL=https://api.openai.com`, - `EMBED_MODEL=text-embedding-3-small`, `EMBED_API_STYLE=openai`, - `EMBED_API_KEY=sk-...`. - -`RERANK_BASE_URL` is optional — unset runs lexical+dense+MMR without the -cross-encoder (`degraded: ["rerank_unavailable"]`, still ranked/citable hits). - -`EMBED_BASE_URL`/`EMBED_MODEL` are optional too — unset both to run -lexical-only (`degraded: ["dense_unavailable", "lexical_only"]`, no dense -channel, `add` still captures documents unvectorized). See the lexical-only -note above. - -## Testing - -`bun test ./src` (`bun run test`), coverage via `bun run test:coverage`. Every -`core/*` module, most services, and the route/identity layer have colocated -`*.test.ts` files exercising pure logic and mocked-boundary behavior — no -external services required. The SDK ships with unit tests only. diff --git a/LICENSE b/LICENSE index f6683e7..c6487f4 100644 --- a/LICENSE +++ b/LICENSE @@ -1,501 +1,176 @@ - GNU LESSER GENERAL PUBLIC LICENSE - Version 2.1, February 1999 - - Copyright (C) 1991, 1999 Free Software Foundation, Inc. - - Everyone is permitted to copy and distribute verbatim copies - of this license document, but changing it is not allowed. - -[This is the first released version of the Lesser GPL. It also counts - as the successor of the GNU Library Public License, version 2, hence - the version number 2.1.] - - Preamble - - The licenses for most software are designed to take away your -freedom to share and change it. By contrast, the GNU General Public -Licenses are intended to guarantee your freedom to share and change -free software--to make sure the software is free for all its users. - - This license, the Lesser General Public License, applies to some -specially designated software packages--typically libraries--of the -Free Software Foundation and other authors who decide to use it. You -can use it too, but we suggest you first think carefully about whether -this license or the ordinary General Public License is the better -strategy to use in any particular case, based on the explanations below. - - When we speak of free software, we are referring to freedom of use, -not price. Our General Public Licenses are designed to make sure that -you have the freedom to distribute copies of free software (and charge -for this service if you wish); that you receive source code or can get -it if you want it; that you can change the software and use pieces of -it in new free programs; and that you are informed that you can do -these things. - - To protect your rights, we need to make restrictions that forbid -distributors to deny you these rights or to ask you to surrender these -rights. These restrictions translate to certain responsibilities for -you if you distribute copies of the library or if you modify it. - - For example, if you distribute copies of the library, whether gratis -or for a fee, you must give the recipients all the rights that we gave -you. You must make sure that they, too, receive or can get the source -code. If you link other code with the library, you must provide -complete object files to the recipients, so that they can relink them -with the library after making changes to the library and recompiling -it. And you must show them these terms so they know their rights. - - We protect your rights with a two-step method: (1) we copyright the -library, and (2) we offer you this license, which gives you legal -permission to copy, distribute and/or modify the library. - - To protect each distributor, we want to make it very clear that -there is no warranty for the free library. Also, if the library is -modified by someone else and passed on, the recipients should know -that what they have is not the original version, so that the original -author's reputation will not be affected by problems that might be -introduced by others. - - Finally, software patents pose a constant threat to the existence of -any free program. We wish to make sure that a company cannot -effectively restrict the users of a free program by obtaining a -restrictive license from a patent holder. Therefore, we insist that -any patent license obtained for a version of the library must be -consistent with the full freedom of use specified in this license. - - Most GNU software, including some libraries, is covered by the -ordinary GNU General Public License. This license, the GNU Lesser -General Public License, applies to certain designated libraries, and -is quite different from the ordinary General Public License. We use -this license for certain libraries in order to permit linking those -libraries into non-free programs. - - When a program is linked with a library, whether statically or using -a shared library, the combination of the two is legally speaking a -combined work, a derivative of the original library. The ordinary -General Public License therefore permits such linking only if the -entire combination fits its criteria of freedom. The Lesser General -Public License permits more lax criteria for linking other code with -the library. - - We call this license the "Lesser" General Public License because it -does Less to protect the user's freedom than the ordinary General -Public License. It also provides other free software developers Less -of an advantage over competing non-free programs. These disadvantages -are the reason we use the ordinary General Public License for many -libraries. However, the Lesser license provides advantages in certain -special circumstances. - - For example, on rare occasions, there may be a special need to -encourage the widest possible use of a certain library, so that it becomes -a de-facto standard. To achieve this, non-free programs must be -allowed to use the library. A more frequent case is that a free -library does the same job as widely used non-free libraries. In this -case, there is little to gain by limiting the free library to free -software only, so we use the Lesser General Public License. - - In other cases, permission to use a particular library in non-free -programs enables a greater number of people to use a large body of -free software. For example, permission to use the GNU C Library in -non-free programs enables many more people to use the whole GNU -operating system, as well as its variant, the GNU/Linux operating -system. - - Although the Lesser General Public License is Less protective of the -users' freedom, it does ensure that the user of a program that is -linked with the Library has the freedom and the wherewithal to run -that program using a modified version of the Library. - - The precise terms and conditions for copying, distribution and -modification follow. Pay close attention to the difference between a -"work based on the library" and a "work that uses the library". The -former contains code derived from the library, whereas the latter must -be combined with the library in order to run. - - GNU LESSER GENERAL PUBLIC LICENSE - TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION - - 0. This License Agreement applies to any software library or other -program which contains a notice placed by the copyright holder or -other authorized party saying it may be distributed under the terms of -this Lesser General Public License (also called "this License"). -Each licensee is addressed as "you". - - A "library" means a collection of software functions and/or data -prepared so as to be conveniently linked with application programs -(which use some of those functions and data) to form executables. - - The "Library", below, refers to any such software library or work -which has been distributed under these terms. A "work based on the -Library" means either the Library or any derivative work under -copyright law: that is to say, a work containing the Library or a -portion of it, either verbatim or with modifications and/or translated -straightforwardly into another language. (Hereinafter, translation is -included without limitation in the term "modification".) - - "Source code" for a work means the preferred form of the work for -making modifications to it. For a library, complete source code means -all the source code for all modules it contains, plus any associated -interface definition files, plus the scripts used to control compilation -and installation of the library. - - Activities other than copying, distribution and modification are not -covered by this License; they are outside its scope. The act of -running a program using the Library is not restricted, and output from -such a program is covered only if its contents constitute a work based -on the Library (independent of the use of the Library in a tool for -writing it). Whether that is true depends on what the Library does -and what the program that uses the Library does. - - 1. You may copy and distribute verbatim copies of the Library's -complete source code as you receive it, in any medium, provided that -you conspicuously and appropriately publish on each copy an -appropriate copyright notice and disclaimer of warranty; keep intact -all the notices that refer to this License and to the absence of any -warranty; and distribute a copy of this License along with the -Library. - - You may charge a fee for the physical act of transferring a copy, -and you may at your option offer warranty protection in exchange for a -fee. - - 2. You may modify your copy or copies of the Library or any portion -of it, thus forming a work based on the Library, and copy and -distribute such modifications or work under the terms of Section 1 -above, provided that you also meet all of these conditions: - - a) The modified work must itself be a software library. - - b) You must cause the files modified to carry prominent notices - stating that you changed the files and the date of any change. - - c) You must cause the whole of the work to be licensed at no - charge to all third parties under the terms of this License. - - d) If a facility in the modified Library refers to a function or a - table of data to be supplied by an application program that uses - the facility, other than as an argument passed when the facility - is invoked, then you must make a good faith effort to ensure that, - in the event an application does not supply such function or - table, the facility still operates, and performs whatever part of - its purpose remains meaningful. - - (For example, a function in a library to compute square roots has - a purpose that is entirely well-defined independent of the - application. Therefore, Subsection 2d requires that any - application-supplied function or table used by this function must - be optional: if the application does not supply it, the square - root function must still compute square roots.) - -These requirements apply to the modified work as a whole. If -identifiable sections of that work are not derived from the Library, -and can be reasonably considered independent and separate works in -themselves, then this License, and its terms, do not apply to those -sections when you distribute them as separate works. But when you -distribute the same sections as part of a whole which is a work based -on the Library, the distribution of the whole must be on the terms of -this License, whose permissions for other licensees extend to the -entire whole, and thus to each and every part regardless of who wrote -it. - -Thus, it is not the intent of this section to claim rights or contest -your rights to work written entirely by you; rather, the intent is to -exercise the right to control the distribution of derivative or -collective works based on the Library. - -In addition, mere aggregation of another work not based on the Library -with the Library (or with a work based on the Library) on a volume of -a storage or distribution medium does not bring the other work under -the scope of this License. - - 3. You may opt to apply the terms of the ordinary GNU General Public -License instead of this License to a given copy of the Library. To do -this, you must alter all the notices that refer to this License, so -that they refer to the ordinary GNU General Public License, version 2, -instead of to this License. (If a newer version than version 2 of the -ordinary GNU General Public License has appeared, then you can specify -that version instead if you wish.) Do not make any other change in -these notices. - - Once this change is made in a given copy, it is irreversible for -that copy, so the ordinary GNU General Public License applies to all -subsequent copies and derivative works made from that copy. - - This option is useful when you wish to copy part of the code of -the Library into a program that is not a library. - - 4. You may copy and distribute the Library (or a portion or -derivative of it, under Section 2) in object code or executable form -under the terms of Sections 1 and 2 above provided that you accompany -it with the complete corresponding machine-readable source code, which -must be distributed under the terms of Sections 1 and 2 above on a -medium customarily used for software interchange. - - If distribution of object code is made by offering access to copy -from a designated place, then offering equivalent access to copy the -source code from the same place satisfies the requirement to -distribute the source code, even though third parties are not -compelled to copy the source along with the object code. - - 5. A program that contains no derivative of any portion of the -Library, but is designed to work with the Library by being compiled or -linked with it, is called a "work that uses the Library". Such a -work, in isolation, is not a derivative work of the Library, and -therefore falls outside the scope of this License. - - However, linking a "work that uses the Library" with the Library -creates an executable that is a derivative of the Library (because it -contains portions of the Library), rather than a "work that uses the -library". The executable is therefore covered by this License. -Section 6 states terms for distribution of such executables. - - When a "work that uses the Library" uses material from a header file -that is part of the Library, the object code for the work may be a -derivative work of the Library even though the source code is not. -Whether this is true is especially significant if the work can be -linked without the Library, or if the work is itself a library. The -threshold for this to be true is not precisely defined by law. - - If such an object file uses only numerical parameters, data -structure layouts and accessors, and small macros and small inline -functions (ten lines or less in length), then the use of the object -file is unrestricted, regardless of whether it is legally a derivative -work. (Executables containing this object code plus portions of the -Library will still fall under Section 6.) - - Otherwise, if the work is a derivative of the Library, you may -distribute the object code for the work under the terms of Section 6. -Any executables containing that work also fall under Section 6, -whether or not they are linked directly with the Library itself. - - 6. As an exception to the Sections above, you may also combine or -link a "work that uses the Library" with the Library to produce a -work containing portions of the Library, and distribute that work -under terms of your choice, provided that the terms permit -modification of the work for the customer's own use and reverse -engineering for debugging such modifications. - - You must give prominent notice with each copy of the work that the -Library is used in it and that the Library and its use are covered by -this License. You must supply a copy of this License. If the work -during execution displays copyright notices, you must include the -copyright notice for the Library among them, as well as a reference -directing the user to the copy of this License. Also, you must do one -of these things: - - a) Accompany the work with the complete corresponding - machine-readable source code for the Library including whatever - changes were used in the work (which must be distributed under - Sections 1 and 2 above); and, if the work is an executable linked - with the Library, with the complete machine-readable "work that - uses the Library", as object code and/or source code, so that the - user can modify the Library and then relink to produce a modified - executable containing the modified Library. (It is understood - that the user who changes the contents of definitions files in the - Library will not necessarily be able to recompile the application - to use the modified definitions.) - - b) Use a suitable shared library mechanism for linking with the - Library. A suitable mechanism is one that (1) uses at run time a - copy of the library already present on the user's computer system, - rather than copying library functions into the executable, and (2) - will operate properly with a modified version of the library, if - the user installs one, as long as the modified version is - interface-compatible with the version that the work was made with. - - c) Accompany the work with a written offer, valid for at - least three years, to give the same user the materials - specified in Subsection 6a, above, for a charge no more - than the cost of performing this distribution. - - d) If distribution of the work is made by offering access to copy - from a designated place, offer equivalent access to copy the above - specified materials from the same place. - - e) Verify that the user has already received a copy of these - materials or that you have already sent this user a copy. - - For an executable, the required form of the "work that uses the -Library" must include any data and utility programs needed for -reproducing the executable from it. However, as a special exception, -the materials to be distributed need not include anything that is -normally distributed (in either source or binary form) with the major -components (compiler, kernel, and so on) of the operating system on -which the executable runs, unless that component itself accompanies -the executable. - - It may happen that this requirement contradicts the license -restrictions of other proprietary libraries that do not normally -accompany the operating system. Such a contradiction means you cannot -use both them and the Library together in an executable that you -distribute. - - 7. You may place library facilities that are a work based on the -Library side-by-side in a single library together with other library -facilities not covered by this License, and distribute such a combined -library, provided that the separate distribution of the work based on -the Library and of the other library facilities is otherwise -permitted, and provided that you do these two things: - - a) Accompany the combined library with a copy of the same work - based on the Library, uncombined with any other library - facilities. This must be distributed under the terms of the - Sections above. - - b) Give prominent notice with the combined library of the fact - that part of it is a work based on the Library, and explaining - where to find the accompanying uncombined form of the same work. - - 8. You may not copy, modify, sublicense, link with, or distribute -the Library except as expressly provided under this License. Any -attempt otherwise to copy, modify, sublicense, link with, or -distribute the Library is void, and will automatically terminate your -rights under this License. However, parties who have received copies, -or rights, from you under this License will not have their licenses -terminated so long as such parties remain in full compliance. - - 9. You are not required to accept this License, since you have not -signed it. However, nothing else grants you permission to modify or -distribute the Library or its derivative works. These actions are -prohibited by law if you do not accept this License. Therefore, by -modifying or distributing the Library (or any work based on the -Library), you indicate your acceptance of this License to do so, and -all its terms and conditions for copying, distributing or modifying -the Library or works based on it. - - 10. Each time you redistribute the Library (or any work based on the -Library), the recipient automatically receives a license from the -original licensor to copy, distribute, link with or modify the Library -subject to these terms and conditions. You may not impose any further -restrictions on the recipients' exercise of the rights granted herein. -You are not responsible for enforcing compliance by third parties with -this License. - - 11. If, as a consequence of a court judgment or allegation of patent -infringement or for any other reason (not limited to patent issues), -conditions are imposed on you (whether by court order, agreement or -otherwise) that contradict the conditions of this License, they do not -excuse you from the conditions of this License. If you cannot -distribute so as to satisfy simultaneously your obligations under this -License and any other pertinent obligations, then as a consequence you -may not distribute the Library at all. For example, if a patent -license would not permit royalty-free redistribution of the Library by -all those who receive copies directly or indirectly through you, then -the only way you could satisfy both it and this License would be to -refrain entirely from distribution of the Library. - -If any portion of this section is held invalid or unenforceable under any -particular circumstance, the balance of the section is intended to apply, -and the section as a whole is intended to apply in other circumstances. - -It is not the purpose of this section to induce you to infringe any -patents or other property right claims or to contest validity of any -such claims; this section has the sole purpose of protecting the -integrity of the free software distribution system which is -implemented by public license practices. Many people have made -generous contributions to the wide range of software distributed -through that system in reliance on consistent application of that -system; it is up to the author/donor to decide if he or she is willing -to distribute software through any other system and a licensee cannot -impose that choice. - -This section is intended to make thoroughly clear what is believed to -be a consequence of the rest of this License. - - 12. If the distribution and/or use of the Library is restricted in -certain countries either by patents or by copyrighted interfaces, the -original copyright holder who places the Library under this License may add -an explicit geographical distribution limitation excluding those countries, -so that distribution is permitted only in or among countries not thus -excluded. In such case, this License incorporates the limitation as if -written in the body of this License. - - 13. The Free Software Foundation may publish revised and/or new -versions of the Lesser General Public License from time to time. -Such new versions will be similar in spirit to the present version, -but may differ in detail to address new problems or concerns. - -Each version is given a distinguishing version number. If the Library -specifies a version number of this License which applies to it and -"any later version", you have the option of following the terms and -conditions either of that version or of any later version published by -the Free Software Foundation. If the Library does not specify a -license version number, you may choose any version ever published by -the Free Software Foundation. - - 14. If you wish to incorporate parts of the Library into other free -programs whose distribution conditions are incompatible with these, -write to the author to ask for permission. For software which is -copyrighted by the Free Software Foundation, write to the Free -Software Foundation; we sometimes make exceptions for this. Our -decision will be guided by the two goals of preserving the free status -of all derivatives of our free software and of promoting the sharing -and reuse of software generally. - - NO WARRANTY - - 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO -WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. -EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR -OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY -KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE -IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR -PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE -LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME -THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. - - 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN -WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY -AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU -FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR -CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE -LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING -RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A -FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF -SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH -DAMAGES. - - END OF TERMS AND CONDITIONS - - How to Apply These Terms to Your New Libraries - - If you develop a new library, and you want it to be of the greatest -possible use to the public, we recommend making it free software that -everyone can redistribute and change. You can do so by permitting -redistribution under these terms (or, alternatively, under the terms of the -ordinary General Public License). - - To apply these terms, attach the following notices to the library. It is -safest to attach them to the start of each source file to most effectively -convey the exclusion of warranty; and each file should have at least the -"copyright" line and a pointer to where the full notice is found. - - - Copyright (C) - - This library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - This library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with this library; if not, see . - -Also add information on how to contact you by electronic and paper mail. - -You should also get your employer (if you work as a programmer) or your -school, if any, to sign a "copyright disclaimer" for the library, if -necessary. Here is a sample; alter the names: - - Yoyodyne, Inc., hereby disclaims all copyright interest in the - library `Frob' (a library for tweaking knobs) written by James Random Hacker. - - , 1 April 1990 - Moe Ghoul, President of Vice +GNU LESSER GENERAL PUBLIC LICENSE +Version 2.1, February 1999 + +Copyright (C) 1991, 1999 Free Software Foundation, Inc. +51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + +Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts as the successor of the GNU Library Public License, version 2, hence the version number 2.1.] + +Preamble + +The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public Licenses are intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. + +This license, the Lesser General Public License, applies to some specially designated software packages--typically libraries--of the Free Software Foundation and other authors who decide to use it. You can use it too, but we suggest you first think carefully about whether this license or the ordinary General Public License is the better strategy to use in any particular case, based on the explanations below. + +When we speak of free software, we are referring to freedom of use, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish); that you receive source code or can get it if you want it; that you can change the software and use pieces of it in new free programs; and that you are informed that you can do these things. + +To protect your rights, we need to make restrictions that forbid distributors to deny you these rights or to ask you to surrender these rights. These restrictions translate to certain responsibilities for you if you distribute copies of the library or if you modify it. + +For example, if you distribute copies of the library, whether gratis or for a fee, you must give the recipients all the rights that we gave you. You must make sure that they, too, receive or can get the source code. If you link other code with the library, you must provide complete object files to the recipients, so that they can relink them with the library after making changes to the library and recompiling it. And you must show them these terms so they know their rights. + +We protect your rights with a two-step method: (1) we copyright the library, and (2) we offer you this license, which gives you legal permission to copy, distribute and/or modify the library. + +To protect each distributor, we want to make it very clear that there is no warranty for the free library. Also, if the library is modified by someone else and passed on, the recipients should know that what they have is not the original version, so that the original author's reputation will not be affected by problems that might be introduced by others. + +Finally, software patents pose a constant threat to the existence of any free program. We wish to make sure that a company cannot effectively restrict the users of a free program by obtaining a restrictive license from a patent holder. Therefore, we insist that any patent license obtained for a version of the library must be consistent with the full freedom of use specified in this license. + +Most GNU software, including some libraries, is covered by the ordinary GNU General Public License. This license, the GNU Lesser General Public License, applies to certain designated libraries, and is quite different from the ordinary General Public License. We use this license for certain libraries in order to permit linking those libraries into non-free programs. + +When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library. + +We call this license the "Lesser" General Public License because it does Less to protect the user's freedom than the ordinary General Public License. It also provides other free software developers Less of an advantage over competing non-free programs. These disadvantages are the reason we use the ordinary General Public License for many libraries. However, the Lesser license provides advantages in certain special circumstances. + +For example, on rare occasions, there may be a special need to encourage the widest possible use of a certain library, so that it becomes a de-facto standard. To achieve this, non-free programs must be allowed to use the library. A more frequent case is that a free library does the same job as widely used non-free libraries. In this case, there is little to gain by limiting the free library to free software only, so we use the Lesser General Public License. + +In other cases, permission to use a particular library in non-free programs enables a greater number of people to use a large body of free software. For example, permission to use the GNU C Library in non-free programs enables many more people to use the whole GNU operating system, as well as its variant, the GNU/Linux operating system. + +Although the Lesser General Public License is Less protective of the users' freedom, it does ensure that the user of a program that is linked with the Library has the freedom and the wherewithal to run that program using a modified version of the Library. + +The precise terms and conditions for copying, distribution and modification follow. Pay close attention to the difference between a "work based on the library" and a "work that uses the library". The former contains code derived from the library, whereas the latter must be combined with the library in order to run. + +GNU LESSER GENERAL PUBLIC LICENSE +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + +0. This License Agreement applies to any software library or other program which contains a notice placed by the copyright holder or other authorized party saying it may be distributed under the terms of this Lesser General Public License (also called "this License"). Each licensee is addressed as "you". + +A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application programs (which use some of those functions and data) to form executables. + +The "Library", below, refers to any such software library or work which has been distributed under these terms. A "work based on the Library" means either the Library or any derivative work under copyright law: that is to say, a work containing the Library or a portion of it, either verbatim or with modifications and/or translated straightforwardly into another language. (Hereinafter, translation is included without limitation in the term "modification".) + +"Source code" for a work means the preferred form of the work for making modifications to it. For a library, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the library. + +Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running a program using the Library is not restricted, and output from such a program is covered only if its contents constitute a work based on the Library (independent of the use of the Library in a tool for writing it). Whether that is true depends on what the Library does and what the program that uses the Library does. + +1. You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and distribute a copy of this License along with the Library. + +You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful. + +(For example, a function in a library to compute square roots has a purpose that is entirely well-defined independent of the application. Therefore, Subsection 2d requires that any application-supplied function or table used by this function must be optional: if the application does not supply it, the square root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Library, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Library, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library with the Library (or with a work based on the Library) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. + +3. You may opt to apply the terms of the ordinary GNU General Public License instead of this License to a given copy of the Library. To do this, you must alter all the notices that refer to this License, so that they refer to the ordinary GNU General Public License, version 2, instead of to this License. (If a newer version than version 2 of the ordinary GNU General Public License has appeared, then you can specify that version instead if you wish.) Do not make any other change in these notices. + +Once this change is made in a given copy, it is irreversible for that copy, so the ordinary GNU General Public License applies to all subsequent copies and derivative works made from that copy. + +This option is useful when you wish to copy part of the code of the Library into a program that is not a library. + +4. You may copy and distribute the Library (or a portion or derivative of it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange. + +If distribution of object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place satisfies the requirement to distribute the source code, even though third parties are not compelled to copy the source along with the object code. + +5. A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License. + +However, linking a "work that uses the Library" with the Library creates an executable that is a derivative of the Library (because it contains portions of the Library), rather than a "work that uses the library". The executable is therefore covered by this License. Section 6 states terms for distribution of such executables. + +When a "work that uses the Library" uses material from a header file that is part of the Library, the object code for the work may be a derivative work of the Library even though the source code is not. Whether this is true is especially significant if the work can be linked without the Library, or if the work is itself a library. The threshold for this to be true is not precisely defined by law. + +If such an object file uses only numerical parameters, data structure layouts and accessors, and small macros and small inline functions (ten lines or less in length), then the use of the object file is unrestricted, regardless of whether it is legally a derivative work. (Executables containing this object code plus portions of the Library will still fall under Section 6.) + +Otherwise, if the work is a derivative of the Library, you may distribute the object code for the work under the terms of Section 6. Any executables containing that work also fall under Section 6, whether or not they are linked directly with the Library itself. + +6. As an exception to the Sections above, you may also combine or link a "work that uses the Library" with the Library to produce a work containing portions of the Library, and distribute that work under terms of your choice, provided that the terms permit modification of the work for the customer's own use and reverse engineering for debugging such modifications. + +You must give prominent notice with each copy of the work that the Library is used in it and that the Library and its use are covered by this License. You must supply a copy of this License. If the work during execution displays copyright notices, you must include the copyright notice for the Library among them, as well as a reference directing the user to the copy of this License. Also, you must do one of these things: + + a) Accompany the work with the complete corresponding machine-readable source code for the Library including whatever changes were used in the work (which must be distributed under Sections 1 and 2 above); and, if the work is an executable linked with the Library, with the complete machine-readable "work that uses the Library", as object code and/or source code, so that the user can modify the Library and then relink to produce a modified executable containing the modified Library. (It is understood that the user who changes the contents of definitions files in the Library will not necessarily be able to recompile the application to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is one that (1) uses at run time a copy of the library already present on the user's computer system, rather than copying library functions into the executable, and (2) will operate properly with a modified version of the library, if the user installs one, as long as the modified version is interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at least three years, to give the same user the materials specified in Subsection 6a, above, for a charge no more than the cost of performing this distribution. + + d) If distribution of the work is made by offering access to copy from a designated place, offer equivalent access to copy the above specified materials from the same place. + + e) Verify that the user has already received a copy of these materials or that you have already sent this user a copy. + +For an executable, the required form of the "work that uses the Library" must include any data and utility programs needed for reproducing the executable from it. However, as a special exception, the materials to be distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. + +It may happen that this requirement contradicts the license restrictions of other proprietary libraries that do not normally accompany the operating system. Such a contradiction means you cannot use both them and the Library together in an executable that you distribute. + +7. You may place library facilities that are a work based on the Library side-by-side in a single library together with other library facilities not covered by this License, and distribute such a combined library, provided that the separate distribution of the work based on the Library and of the other library facilities is otherwise permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work based on the Library, uncombined with any other library facilities. This must be distributed under the terms of the Sections above. + + b) Give prominent notice with the combined library of the fact that part of it is a work based on the Library, and explaining where to find the accompanying uncombined form of the same work. + +8. You may not copy, modify, sublicense, link with, or distribute the Library except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, link with, or distribute the Library is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. + +9. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Library or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Library (or any work based on the Library), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Library or works based on it. + +10. Each time you redistribute the Library (or any work based on the Library), the recipient automatically receives a license from the original licensor to copy, distribute, link with or modify the Library subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties with this License. + +11. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Library at all. For example, if a patent license would not permit royalty-free redistribution of the Library by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply, and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. + +This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. + +12. If the distribution and/or use of the Library is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Library under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. + +13. The Free Software Foundation may publish revised and/or new versions of the Lesser General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Library does not specify a license version number, you may choose any version ever published by the Free Software Foundation. + +14. If you wish to incorporate parts of the Library into other free programs whose distribution conditions are incompatible with these, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. + +NO WARRANTY + +15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + +16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS + +How to Apply These Terms to Your New Libraries + +If you develop a new library, and you want it to be of the greatest possible use to the public, we recommend making it free software that everyone can redistribute and change. You can do so by permitting redistribution under these terms (or, alternatively, under the terms of the ordinary General Public License). + +To apply these terms, attach the following notices to the library. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. + + one line to give the library's name and an idea of what it does. + Copyright (C) year name of author + + This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Also add information on how to contact you by electronic and paper mail. + +You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the library, if necessary. Here is a sample; alter the names: + +Yoyodyne, Inc., hereby disclaims all copyright interest in +the library `Frob' (a library for tweaking knobs) written +by James Random Hacker. + +signature of Ty Coon, 1 April 1990 +Ty Coon, President of Vice That's all there is to it! diff --git a/PRODUCT.md b/PRODUCT.md deleted file mode 100644 index ac5ee00..0000000 --- a/PRODUCT.md +++ /dev/null @@ -1,165 +0,0 @@ -# Corbits Memory — Product shape - -Memory for Interchange hubs: durable documents, hybrid search, recent list. - -**You mount it on the hub (~5 lines). That exposes protected routes. Agents -and ingestion modules call those routes.** Capture (`add`) writes; search -retrieves. Workbench and coding agents are clients — not owners of auth. -Inference stays host-injected. Deployed agents do **not** git-install this -package: they carry `@corbits/memory/sidecar-bundle` and hit the parallel -`mountWorkflowMemory` routes. - -## Default pipeline (locked) - -```text -add → ingest elements → process (optional) -``` - -| Stage | Meaning | Where | -| ------------------- | --------------------------------------------------------------- | --------------------------------------------------------------------- | -| **add** | Something arrives (agent tool, host job, webhook body) | Caller → `memory.add` / `POST …/memory/add` | -| **ingest elements** | Normalize → raw capture → chunks / edges → embed → search-ready | Default `DocumentStore` capture path (sync on `add`) | -| **process** | Optional brain work: classify, claims, links, forget | Host workflow / injected inference — same run as ingest when possible | - -Preferred host shape: **one ingest workflow** receives the event, calls `add` -(ingest elements), then runs process steps in the same body (or a child step). -No pull feed required on that path — the workflow already has the payload. - -**Pull feed + resident distiller** are optional: multi-writer backfill, replay, -or polish when other code also `add`s outside the ingest workflow. See -`docs/DISTILLER.md` and `docs/FEED.md`. - -## Shape (locked) - -**`src/` is the `@corbits/memory` SDK.** Interchange is the hub — the SDK -never creates one; it mounts onto yours. - -| Surface | Role | -| -------------------------------------------------------- | --------------------------------------------------------------------- | -| `createMemory({ … })` | Build the plane | -| `createMemoryRoutes({ memory, requireGrant })` | Hono sub-app the host mounts at `/api/tenants/:tenantId/memory` | -| `mountWorkflowMemory(app, { memory, agentToken })` | Parallel run-scoped `/api/workflow-memory/*` for deployed agents | -| `loadMemoryConfig()` | Config from env | -| `runMemoryMigrations(dbConfig, { schema, ftsLanguage })` | Apply pgvector schema | -| `@corbits/memory/sidecar-bundle` | Deployed-agent factory — no client code, no base URL, no token | -| `@corbits/memory/distiller` | Optional process helpers: `runDistillTick`, `createResidentDistiller` | - -### Verbs - -| Method | HTTP | Grant | Meaning | -| -------- | ------------------------------------------- | --------------- | ----------------------------------------------------------------------------------- | -| `add` | `POST /api/tenants/:tenantId/memory/add` | `memory:add` | Ingest: capture + derive (chunk/embed on default store) | -| `search` | `POST /api/tenants/:tenantId/memory/search` | `memory:search` | Hybrid retrieval (+ optional live sources); hits may include additive `attribution` | -| `list` | `GET /api/tenants/:tenantId/memory/list` | `memory:search` | Recent documents for the principal | -| `feed` | `GET /api/tenants/:tenantId/memory/feed` | `memory:search` | Cursor pull of new live versions (optional multi-writer / backfill) | - -Engine-only plane helpers (no HTTP yet): transform/replay, retention -(`deprecateVersion` / `tombstoneDocument` / … — see `docs/RETENTION.md`), -share-grant materialization. Process helpers: -`createResidentDistiller` / `runDistillTick` (`docs/DISTILLER.md`) — host -injects inference; not the default ingest path. - -Identity is always **`principalId` + `tenantId`** on the plane. Tenant HTTP -routes never take body identity — they read `c.get("principal")` from -Interchange context. Run-scoped sidecar routes read the verified workflow -run (`agentToken.verify` + `x-workflow-run-address`); the sidecar factory -never names a host or carries a token. - -### How it is used - -``` -Agent / host ingest workflow - │ tool call or host worker - │ → POST|GET /api/tenants/:tenantId/memory/* - │ authenticated by Interchange (session | API key | MCP OAuth) - │ -Deployed agent (sidecar-bundle) - │ hub credential + run address - │ → POST|GET /api/workflow-memory/* (mountWorkflowMemory) - ▼ -┌──────────────────────────────────────────────┐ -│ Host Interchange createApp │ -│ principal + tenant on context │ -│ + createMemory({ grantStore, … }) │ -│ + app.route(…, createMemoryRoutes(deps)) │ -│ grants: memory:add | memory:search │ -│ documentStore: pgvector | host | fake │ -│ + mountWorkflowMemory(app, { memory, … }) │ -│ │ in-process │ -│ ▼ │ -│ Memory plane: add (capture) / search / list │ -│ → DocumentStore (sole durable backend) │ -└──────────────────────────────────────────────┘ -``` - -1. **Mount** — host passes `app` + the same grant store it already uses. -2. **Sidecar (deployed agents)** — agents carry - `@corbits/memory/sidecar-bundle`. It holds no client code, no base URL, - and no token: it resolves the host `hub` credential and calls - `/api/workflow-memory/*`. Host wires `mountWorkflowMemory` in parallel - with the tenant routes (see README How it works). Not the primary - install — that is still `createMemory` + `loadMemoryConfig`. -3. **Ingestion** — preferred: one host workflow (or module) does - **add → ingest elements (capture) → process**. Mechanical capture is - inside `add` on the default store; process (claims / links) is - host-injected inference in the same pipeline when you want a company - brain. - -### Ports - -| Port | Purpose | -| ---------------- | -------------------------------------------------------------------------------------------------------------------- | -| `DocumentStore` | Sole durable backend for add/search/list (default: pgvector). Inject fakes or a host/adapter store to skip Postgres. | -| `SourceProvider` | Optional live search merge (fail-soft). Not a store replacement. | - -Optional sibling packages (not in this tree): Mem0 / Supermemory document -stores, Linear tools. Core never imports vendor SDKs. - -### What is not in scope - -- No auth, API keys, OAuth, webhooks, SPA, or standalone server in core. -- No answer/generation endpoint — host owns inference. -- Workbench is a client, not required. -- Core does not run the ingest workflow process — the host does. -- Deployed agents do not git-install this package as a sidecar and do not - receive a memory base URL or token — that is the sidecar-bundle contract. - -**Default durable store:** Postgres via `DATABASE_URL`, tables under the -**`memory`** schema. When -`documentStore` is injected, Postgres is not opened. Cross-refs are plain -`text` — no FKs into the host control plane. - -## Identity and access (one authz system) - -Memory does not ship a second ACL. Document access uses the host’s -`@intx/authz` grant store. - -1. **Capability** — may this principal use memory? - `authorize(…, resource: "memory", action: "add" | "search")`. -2. **Document access** — each document carries **`accessTags`**. A principal - sees a document if they are the creator **or** - `authorize(…, resource: , action: "search")` allows for any tag. -3. **Share sugars on add** — mint tags only (owner / tenant / peers / - explicit). Host must still grant peers `search` on the relevant tags. - See `docs/AUTHZ-DOCUMENT-ACCESS.md`. - -Default add is **owner-only**. Deny is absence of allow — not a document -block list. - -### On the wire - -```http -POST /api/tenants/:tenantId/memory/add { "title", "text", "access_tags"?, "share"? } -POST /api/tenants/:tenantId/memory/search { "query", "limit?", "kinds"?, "entity_ids"?, "sources"?, "includeEvidence"? } -GET /api/tenants/:tenantId/memory/list ?limit= -``` - -### Live sources - -Local documents are grant-tagged. Live `SourceProvider` hits merge as -enrichment under the host’s connector tokens (fail-soft, no grant tags). - -## Out of scope forever here - -Auth, OAuth for Linear, third-party account management, embedding models -in-process, and any standalone process entrypoint. diff --git a/README.md b/README.md index bcbd25b..599b336 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@ # @corbits/memory -[![npm](https://img.shields.io/npm/v/@corbits/memory.svg)](https://www.npmjs.com/package/@corbits/memory) [![License: LGPL-2.1](https://img.shields.io/badge/license-LGPL--2.1-green.svg)](https://github.com/corbitsdev/corbits-memory/blob/main/LICENSE) - Hybrid semantic and full-text document memory in Postgres with pgvector, with optional embedding and rerank endpoints. A Corbits hub module: it mounts Hono routes on `@intx/hub-api` that check Interchange grants (permissions a principal, a user or agent account, holds on a resource), and ships agent tools for the sidecar, the Interchange agent runtime. ## Why @corbits/memory? @@ -200,8 +198,6 @@ export function buildAssistant(sources: readonly InferencePreference[]) { - `@intx/*`, `drizzle-orm`, `hono`, `hono-openapi` and `postgres` are peer dependencies. - Internal helpers and test fakes are no longer exported from the package root: the share-grant, transform, retention and feed services, the distiller re-exports (import them from `@corbits/memory/distiller`), corroboration scoring, the embed model registry, degrade metrics, the FTS language helpers, `createFakeDocumentStore`/`createFakeSourceProvider`, `createInMemoryWritableGrantStore`/`isWritableGrantStore`, `resolveGrantConfig`/`GrantConfig` and `registerMemoryRoutes`. `runMemoryMigrations` is only exported from `@corbits/memory/migrations`. -See the [changelog](https://github.com/corbitsdev/corbits-memory/blob/main/CHANGELOG.md) for the full list. - ## License [LGPL-2.1-only](https://github.com/corbitsdev/corbits-memory/blob/main/LICENSE) diff --git a/docs/AUTHZ-DOCUMENT-ACCESS.md b/docs/AUTHZ-DOCUMENT-ACCESS.md index 8317b8f..b7c0fe9 100644 --- a/docs/AUTHZ-DOCUMENT-ACCESS.md +++ b/docs/AUTHZ-DOCUMENT-ACCESS.md @@ -193,14 +193,3 @@ Fresh databases apply the baseline migrations (`0001_extensions.sql` + - Per-chunk ACL. - Live channel grant tags (host policy). - Re-implementing roles inside this package. - -## Acceptance - -1. No public plane/HTTP API accepts `visibility` mode or block list as security. -2. Default add is owner-visible only (creator + owner tag). -3. Principal B sees A’s doc only when host grant allows `search` on a tag present on the doc (or B is creator). -4. Capability `memory`/`search` still required for search/list. -5. PRODUCT.md / README describe grant tags, not mini-ACL. - -6. Engine + fakes enforce the algorithm; vendor adapters document principal-bucket limit. -7. `bun run typecheck && bun run test` green. diff --git a/docs/DISTILLER.md b/docs/DISTILLER.md deleted file mode 100644 index c65b887..0000000 --- a/docs/DISTILLER.md +++ /dev/null @@ -1,122 +0,0 @@ -# Process helpers (distiller) - -Default product path is **add → ingest elements → process** in one host -pipeline (`PRODUCT.md`). Mechanical ingest (raw → chunk → embed) already runs -inside `memory.add` on the default store. - -This package’s distiller exports are **optional process helpers** for when -you need LLM claim extraction **outside** that single pipeline: - -- multi-writer: other agents also `add` and you want a backfill worker -- replay / catch-up over a cursor -- fail-soft polish decoupled from the write that ingested the raw note - -They are **not** the primary “how memory is ingested” story. - -## Preferred: process in the same ingest workflow - -```text -onTrigger / host job - 1. receive source event (or fetch) - 2. memory_add // ingest elements - 3. process (optional) // claims / links — same body or child step -``` - -No feed cursor required: the workflow already has the payload. Host injects -inference; tools only need `memory_add` / `memory_search` (and grants). - -Helpers still useful in-process: - -| Export | Use | -| ----------------------------- | -------------------------------------------------------------------------- | -| `buildDistilledClaim` | Wire body with `generator_agent_id`, `provenance=inferred`, `derived_from` | -| `RESIDENT_DISTILLER_AGENT_ID` | Stable generator id if you write claims | - -## Optional: multi-writer / backfill (`runDistillTick`) - -When other writers also `add`, drain new versions with the capture feed: - -```ts -import { runDistillTick } from "@corbits/memory/distiller"; -import { createMemoryHttpClient } from "@corbits/memory"; - -const client = createMemoryHttpClient({ - baseUrl: process.env.MEMORY_BASE_URL!, - tenantId: process.env.MEMORY_TENANT_ID!, - authToken: process.env.MEMORY_AUTH_TOKEN!, -}); - -let cursor = 0; -const result = await runDistillTick({ - client, - after: cursor, - distill: async (entry) => { - // call your model — return skip | poison | write - return { - action: "write", - title: "Claim", - text: "…", - temporalClass: "lesson", - }; - }, -}); -cursor = result.nextCursor; // persist -``` - -Inference is **always injected** (`distill` callback or host agent sources). -The package never embeds a model. - -## Optional: mail-triggered workflow scaffold (`createResidentDistiller`) - -Scaffold for hosts that still want a deployed agent with memory tools + -system prompt (loop-safety, access-tag copy). Prefer wiring **process next to -add** in your ingest workflow; use this for backfill-style residency only. - -```ts -import { createResidentDistiller } from "@corbits/memory/distiller"; - -const { workflow, generatorAgentId } = createResidentDistiller({ - mailTo: "resident-distiller@tenant.example.com", - inference: { - sources: [{ provider: "openai", model: "gpt-4.1-mini" }], - }, -}); -// Deploy only if you need a multi-writer pull consumer — not default ingest. -``` - -The run fires on mail to `mailTo`. The host decides when: mail that address -on a schedule, e.g. from `@corbits/cron`. This package owns no clock. - -## Substrate (plane) - -| Piece | Where | -| -------------------------- | ------------------------------------------------------------------- | -| Ingest on add | capture path — raw + chunks + embed | -| Capture feed (cursor) | `memory.feed` — [FEED.md](./FEED.md) (backfill / multi-writer) | -| Claim identity on add | `generator_agent_id`, `provenance`, `lineage_class`, `derived_from` | -| Wire attribution on search | `SearchItem.attribution` | -| Retention / forgetting | [RETENTION.md](./RETENTION.md) | -| Tools | `@corbits/memory/sidecar-bundle` | - -## Grant manifest (process principal) - -Installer discovery (not live grants): - -- `package.json` → `interchange.grantRequirements` (the single source) -- loaded in-process as `MEMORY_GRANT_REQUIREMENTS` / `MEMORY_CAPABILITY_IDS` - from `@corbits/memory` - -Minimum capabilities: - -- `memory:add` (claim or note writes) -- `memory:search` (corroboration + feed if using backfill) - -Deploy materializes these onto the workflow principal. Copy `accessTags` from -the source onto claim writes — never mint broader tags. - -## Out of scope - -- Host deploy pipeline / secrets -- Push outbox (optional later; not required if process is in-pipeline) -- Core-owned ingest workflow process (host owns that) -- Automatic supports/contradicts edge minting beyond `derived_from` on add diff --git a/docs/FEED.md b/docs/FEED.md index 5ec7256..4393650 100644 --- a/docs/FEED.md +++ b/docs/FEED.md @@ -1,10 +1,10 @@ # Capture feed Stateless, cursorable pull of new **versions** for **optional** multi-writer -backfill / process workers (CL-5868). +backfill / process workers. **Default product path does not need this.** Prefer -**add → ingest elements → process** in one host pipeline (`PRODUCT.md`): the +**add → ingest elements → process** in one host pipeline: the workflow already has the payload, so no pull cursor. Use the feed when: @@ -13,7 +13,7 @@ Use the feed when: - you need catch-up / replay over a durable ordering key - process is intentionally decoupled from the write (fail-soft polish) -## Phase 1 — pull (implemented) +## Pull ``` memory.feed({ tenantId, principalId, after?, limit?, excludeGenerator? }) @@ -37,13 +37,6 @@ Cursor storage is the **consumer's** job (workflow run state). HTTP: `GET /api/tenants/:tenantId/memory/feed?after=&limit=&exclude_generator=` -## Phase 2 — push (design only) - -Post-commit outbox row keyed by `feed_seq` + host dispatcher that mails the -deployment address with version ids. **Not implemented** in core. Phase 1 -`feed_seq` is the ordering key so Phase 2 is additive. Only relevant if process -stays out-of-band from the writer. - ## Non-goals - In-core cron or push dispatcher diff --git a/docs/RETENTION.md b/docs/RETENTION.md index 2fa5623..642b4ad 100644 --- a/docs/RETENTION.md +++ b/docs/RETENTION.md @@ -1,4 +1,4 @@ -# Retention classes (CL-5871) +# Retention classes Versions carry a **retention class** orthogonal to temporal ranking class (`temporal_class`) and lineage (`source_class` / `provenance`). @@ -31,7 +31,7 @@ verb — TTL never hard-deletes. Service module: `src/services/retention.ts`. -## HTTP surface (CL-6288) +## HTTP surface | Route | Grant action | Plane verb | | --------------------------------------------------- | --------------- | -------------------- | @@ -72,4 +72,4 @@ something a single user requests about their own data, and it has no natural per-caller grant (it does not take a `principalId` and touches every matching row tenant-wide). A host that wants it schedules a cron job calling `memory.sweepEphemeral({ tenantId })` in-process (the returned `Memory` -already exposes it); the engine stays cron-free per `ARCHITECTURE.md`. +already exposes it); the engine stays cron-free. diff --git a/e2e/add-search.test.ts b/e2e/add-search.test.ts index 182a1eb..4bb9aa3 100644 --- a/e2e/add-search.test.ts +++ b/e2e/add-search.test.ts @@ -3,7 +3,7 @@ import { createInMemoryGrantStore } from "@intx/authz"; import type { Hono } from "hono"; import type { TenantEnv } from "@intx/hub-api"; -import type { Memory } from "../src/memory.ts"; +import type { Memory } from "../src/memory.js"; import { allow, createTestApp, @@ -12,7 +12,7 @@ import { seedPrincipal, testDatabaseUrl, type TestDb, -} from "./helpers.ts"; +} from "./helpers.js"; describe.skipIf(testDatabaseUrl() === undefined)("add and search", () => { let db: TestDb; diff --git a/e2e/caller-resolver.test.ts b/e2e/caller-resolver.test.ts index a4f8387..fa148e3 100644 --- a/e2e/caller-resolver.test.ts +++ b/e2e/caller-resolver.test.ts @@ -3,11 +3,11 @@ import { createInMemoryGrantStore, type GrantStore } from "@intx/authz"; import { createRequireGrant, type TenantEnv } from "@intx/hub-api"; import { Hono } from "hono"; -import type { Memory } from "../src/memory.ts"; +import type { Memory } from "../src/memory.js"; import { createMemoryRoutes, type CallerResolver, -} from "../src/routes/mount.ts"; +} from "../src/routes/mount.js"; import { allow, createTestDb, @@ -15,7 +15,7 @@ import { seedPrincipal, testDatabaseUrl, type TestDb, -} from "./helpers.ts"; +} from "./helpers.js"; // A machine caller (e.g. a workflow run) never passes the host's session // middleware; the host's callerResolver names its tenant and principal. diff --git a/e2e/context-rows.test.ts b/e2e/context-rows.test.ts index 2e28cf9..59213e9 100644 --- a/e2e/context-rows.test.ts +++ b/e2e/context-rows.test.ts @@ -3,8 +3,8 @@ import { createInMemoryGrantStore } from "@intx/authz"; import { createRequireGrant, type TenantEnv } from "@intx/hub-api"; import { Hono } from "hono"; -import type { Memory } from "../src/memory.ts"; -import { createMemoryRoutes } from "../src/routes/mount.ts"; +import type { Memory } from "../src/memory.js"; +import { createMemoryRoutes } from "../src/routes/mount.js"; import { allow, createTestDb, @@ -12,7 +12,7 @@ import { seedPrincipal, testDatabaseUrl, type TestDb, -} from "./helpers.ts"; +} from "./helpers.js"; describe.skipIf(testDatabaseUrl() === undefined)( "synthesized context rows", diff --git a/e2e/distill-tick.test.ts b/e2e/distill-tick.test.ts index b844758..f82e599 100644 --- a/e2e/distill-tick.test.ts +++ b/e2e/distill-tick.test.ts @@ -1,9 +1,9 @@ import { afterAll, beforeAll, describe, expect, test } from "bun:test"; import { createInMemoryGrantStore } from "@intx/authz"; -import { runDistillTick } from "../src/distiller/tick.ts"; -import { createMemoryHttpClient } from "../src/http-client.ts"; -import type { Memory } from "../src/memory.ts"; +import { runDistillTick } from "../src/distiller/tick.js"; +import { createMemoryHttpClient } from "../src/http-client.js"; +import type { Memory } from "../src/memory.js"; import { allow, createTestApp, @@ -12,7 +12,7 @@ import { seedPrincipal, testDatabaseUrl, type TestDb, -} from "./helpers.ts"; +} from "./helpers.js"; describe.skipIf(testDatabaseUrl() === undefined)("distill tick", () => { let db: TestDb; diff --git a/e2e/embed-client.test.ts b/e2e/embed-client.test.ts index e9efbc2..90ab5e1 100644 --- a/e2e/embed-client.test.ts +++ b/e2e/embed-client.test.ts @@ -5,8 +5,8 @@ import { EmbedTimeoutError, embedTexts, probeEmbedDims, -} from "../src/core/embed-client.ts"; -import { startHttpStub } from "./helpers.ts"; +} from "../src/core/embed-client.js"; +import { startHttpStub } from "./helpers.js"; const stub = startHttpStub(); afterAll(() => stub.stop()); diff --git a/e2e/grants.test.ts b/e2e/grants.test.ts index 63244dd..2b4ce6b 100644 --- a/e2e/grants.test.ts +++ b/e2e/grants.test.ts @@ -3,7 +3,7 @@ import { createInMemoryGrantStore } from "@intx/authz"; import type { Hono } from "hono"; import type { TenantEnv } from "@intx/hub-api"; -import type { Memory } from "../src/memory.ts"; +import type { Memory } from "../src/memory.js"; import { allow, createTestApp, @@ -12,7 +12,7 @@ import { seedPrincipal, testDatabaseUrl, type TestDb, -} from "./helpers.ts"; +} from "./helpers.js"; describe.skipIf(testDatabaseUrl() === undefined)( "grants, forget and purge", diff --git a/e2e/helpers.ts b/e2e/helpers.ts index 53d0f41..f195c7e 100644 --- a/e2e/helpers.ts +++ b/e2e/helpers.ts @@ -9,10 +9,10 @@ import { createRequireGrant, type TenantEnv } from "@intx/hub-api"; import { Hono } from "hono"; import postgres from "postgres"; -import { createMemory, type Memory } from "../src/memory.ts"; -import type { MemoryConfig } from "../src/mount-config.ts"; -import { runMemoryMigrations } from "../src/migrations.ts"; -import { createMemoryRoutes } from "../src/routes/mount.ts"; +import { createMemory, type Memory } from "../src/memory.js"; +import type { MemoryConfig } from "../src/mount-config.js"; +import { runMemoryMigrations } from "../src/migrations.js"; +import { createMemoryRoutes } from "../src/routes/mount.js"; const FTS_LANGUAGE = "english"; diff --git a/e2e/migrations.test.ts b/e2e/migrations.test.ts index 804aaa6..5d33a32 100644 --- a/e2e/migrations.test.ts +++ b/e2e/migrations.test.ts @@ -3,8 +3,8 @@ import { readdir, readFile } from "node:fs/promises"; import { join } from "node:path"; import { runMigrations } from "@intx/db"; -import { runMemoryMigrations } from "../src/migrations.ts"; -import { createEmptyDb, testDatabaseUrl, type TestDb } from "./helpers.ts"; +import { runMemoryMigrations } from "../src/migrations.js"; +import { createEmptyDb, testDatabaseUrl, type TestDb } from "./helpers.js"; const options = { schema: "public", ftsLanguage: "english" }; diff --git a/e2e/plane.test.ts b/e2e/plane.test.ts index 6a98f70..0d9bdb5 100644 --- a/e2e/plane.test.ts +++ b/e2e/plane.test.ts @@ -1,13 +1,13 @@ import { afterAll, beforeAll, describe, expect, test } from "bun:test"; import { createInMemoryGrantStore } from "@intx/authz"; -import { RerankConfigError } from "../src/core/rerank-client.ts"; +import { RerankConfigError } from "../src/core/rerank-client.js"; import { createMemory, MemoryError, type Memory, type MemoryAddParams, -} from "../src/memory.ts"; +} from "../src/memory.js"; import { createTestDb, createTestMemory, @@ -15,7 +15,7 @@ import { testDatabaseUrl, testMemoryConfig, type TestDb, -} from "./helpers.ts"; +} from "./helpers.js"; async function rejection(run: Promise): Promise { const err = await run.then( diff --git a/e2e/rerank-client.test.ts b/e2e/rerank-client.test.ts index e5faf50..3419e29 100644 --- a/e2e/rerank-client.test.ts +++ b/e2e/rerank-client.test.ts @@ -11,8 +11,8 @@ import { defaultMaxDocCharsForModel, rerankDocuments, validateRerankConfig, -} from "../src/core/rerank-client.ts"; -import { startHttpStub } from "./helpers.ts"; +} from "../src/core/rerank-client.js"; +import { startHttpStub } from "./helpers.js"; // .env.example ships RERANK_MODEL=bge-reranker-base with RERANK_MAX_DOC_CHARS // unset; that combination must validate. diff --git a/e2e/upgrade-from-0.1.0.test.ts b/e2e/upgrade-from-0.1.0.test.ts index dbeb6bb..8c89b6a 100644 --- a/e2e/upgrade-from-0.1.0.test.ts +++ b/e2e/upgrade-from-0.1.0.test.ts @@ -5,7 +5,7 @@ import { createInMemoryGrantStore } from "@intx/authz"; import { runMigrations } from "@intx/db"; import * as v010 from "@corbits/memory-0.1.0"; -import { runMemoryMigrations } from "../src/migrations.ts"; +import { runMemoryMigrations } from "../src/migrations.js"; import { allow, createEmptyDb, @@ -14,7 +14,7 @@ import { testDatabaseUrl, testMemoryConfig, type TestDb, -} from "./helpers.ts"; +} from "./helpers.js"; const DOCS = [ { diff --git a/package.json b/package.json index 85ed83c..66e9d4a 100644 --- a/package.json +++ b/package.json @@ -3,9 +3,11 @@ "version": "0.2.0", "description": "Mountable memory add/search/list SDK for Interchange hubs — includes resident distiller", "keywords": [ + "corbits", "hono", "interchange", "knowledge", + "memory", "pgvector", "rag", "search" @@ -28,8 +30,6 @@ ], "type": "module", "sideEffects": false, - "main": "./dist/index.js", - "types": "./dist/index.d.ts", "exports": { ".": { "types": "./dist/index.d.ts", @@ -56,16 +56,17 @@ "access": "public" }, "scripts": { - "db:setup": "bun run scripts/db-setup.ts", - "typecheck": "tsc --noEmit", - "build": "tsc -p tsconfig.build.json", + "build": "rm -rf dist && tsc -p tsconfig.build.json", "prepack": "bun run build", - "test": "bun test ./src", - "test:e2e": "bun test ./e2e", - "test:coverage": "bun test --coverage --coverage-reporter=lcov --coverage-reporter=text ./src ./e2e", + "typecheck": "tsc --noEmit", "lint": "oxlint", "format": "oxfmt", - "format:check": "oxfmt --check" + "format:check": "oxfmt --check", + "test": "bun test src --pass-with-no-tests", + "test:e2e": "bun test e2e", + "check": "bun run typecheck && bun run lint && bun run format:check && bun run test", + "db:setup": "bun run scripts/db-setup.ts", + "test:coverage": "bun test --coverage --coverage-reporter=lcov --coverage-reporter=text ./src ./e2e" }, "dependencies": { "arktype": "^2.1.29" @@ -101,6 +102,10 @@ "hono-openapi": "^1.3.1", "postgres": "^3.4.9" }, + "engines": { + "bun": ">=1.2.0", + "node": ">=24" + }, "interchange": { "grantRequirements": [ { diff --git a/src/core/embed-model-registry.ts b/src/core/embed-model-registry.ts index 8c60687..7a1c633 100644 --- a/src/core/embed-model-registry.ts +++ b/src/core/embed-model-registry.ts @@ -149,8 +149,10 @@ export async function ensureEmbedModel( // lives in the schema: a hard delete cascades document -> version -> // chunk -> embedding with no application cleanup to forget. CREATE TABLE // IF NOT EXISTS cannot retrofit the FK onto a pre-existing table — that - // is a deliberate new-tables-only choice (see IMPLEMENTATION.md for the - // one-time ALTER). + // is a deliberate new-tables-only choice. A pre-existing table takes it + // once: ALTER TABLE memory_embedding_ ADD CONSTRAINT + // memory_embedding__chunk_fk FOREIGN KEY (chunk_id) + // REFERENCES memory_chunk (id) ON DELETE CASCADE. await client.query( `CREATE TABLE IF NOT EXISTS ${tableName} ( chunk_id text PRIMARY KEY, diff --git a/src/grant-requirements.ts b/src/grant-requirements.ts index bff05ab..1aa7f9e 100644 --- a/src/grant-requirements.ts +++ b/src/grant-requirements.ts @@ -23,8 +23,7 @@ const GrantRequirement = type({ * `memory:forget` scoped to what it creates"). **Nothing in this package * reads or enforces this value** — whether a specific caller may actually * forget/purge a specific document is decided entirely by the imperative - * creator check in `services/retention-ownership.ts`. See ARCHITECTURE.md - * § Boundaries for the two-mechanism split. + * creator check in `services/retention-ownership.ts`. */ installHint: "'tenant' | 'creator' | 'invoker'", /** Package surfaces that need the requirement when installed. */ diff --git a/tsconfig.build.json b/tsconfig.build.json index 13b956e..af6694f 100644 --- a/tsconfig.build.json +++ b/tsconfig.build.json @@ -1,20 +1,15 @@ { - // Publish build: emits dist/ (js + d.ts) for the packed tarball. - // Source stays the dev surface (bun test, typecheck); dist is what - // the `default`/`types` export conditions resolve to once packed. "extends": "./tsconfig.json", "compilerOptions": { - "module": "NodeNext", - "moduleResolution": "NodeNext", - "allowImportingTsExtensions": false, "noEmit": false, "declaration": true, "declarationMap": false, "sourceMap": false, + "module": "NodeNext", + "moduleResolution": "NodeNext", "outDir": "dist", - "rootDir": "src", - "composite": false + "rootDir": "src" }, "include": ["src"], - "exclude": ["src/**/*.test.ts", "dist", "node_modules"] + "exclude": ["src/**/*.test.ts"] } diff --git a/tsconfig.json b/tsconfig.json index 5aac854..dda942c 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -1,20 +1,21 @@ { "compilerOptions": { - "lib": ["ESNext"], + "esModuleInterop": true, + "skipLibCheck": true, "target": "ESNext", "module": "ESNext", "moduleResolution": "bundler", "moduleDetection": "force", - "allowImportingTsExtensions": true, + "isolatedModules": true, "verbatimModuleSyntax": true, - "noEmit": true, + "resolveJsonModule": true, "strict": true, - "skipLibCheck": true, "noUncheckedIndexedAccess": true, + "noImplicitOverride": true, "exactOptionalPropertyTypes": true, - "noFallthroughCasesInSwitch": true, - "forceConsistentCasingInFileNames": true, + "noEmit": true, + "lib": ["ESNext"], "types": ["bun"] }, - "include": ["src", "scripts", "e2e"] + "include": ["src", "e2e"] }