Skip to content

Cloud-back the knowledge catalog + cutover runbook (PURE REMOTE) - #14

Closed
andrei-hasna wants to merge 3 commits into
mainfrom
oss-cloud-runtime/knowledge-cloud-catalog-cutover
Closed

Cloud-back the knowledge catalog + cutover runbook (PURE REMOTE)#14
andrei-hasna wants to merge 3 commits into
mainfrom
oss-cloud-runtime/knowledge-cloud-catalog-cutover

Conversation

@andrei-hasna

Copy link
Copy Markdown
Contributor

Follow-up to #12. Completes the knowledge cloud cutover: the CLI catalog now round-trips to the cloud Postgres in cloud mode, plus the data-cutover tooling and runbook.

What

  • knowledge_items table (additive migration + indexes) — the catalog's cloud home. Local mode still uses the JSON store.
  • src/db/cloud-catalog.ts — list/get/count/upsert/add via the vendored storage kit (createKnowledgeCloudClient). PURE REMOTE (Amendment A1): every call round-trips, no cache/mirror.
  • CLI wiringadd/list/get route to cloud when HASNA_KNOWLEDGE_STORAGE_MODE=cloud. Remaining catalog mutators (update/archive/restore/untag/delete/upsert/prune/dedupe/export/stats/inventory) are guarded in cloud mode so they never silently operate on the local JSON store — follow-up to wire them.
  • scripts/import-cloud-catalog.mjs — idempotent JSONL importer (upsert by id).
  • docs/migration/local-to-cloud-cutover.md — full runbook incl. the default-mode flip step (env/fleet-config, reversible, no code release) and the tunnel sslmode=requireuselibpqcompat=true gotcha.

Verified live (shared RDS via SSM tunnel, S3 prod)

  • Schema applied through apply-cloud-migrations.mjs; idempotent (64/64, 0 pending).
  • Imported the 48-record catalog export → knowledge_items count 0→48 (47 active + 1 archived, matches source).
  • Read back through the real CLI code path with the least-privilege app role (knowledge_app): knowledge list --include-archived total=48, knowledge get round-trips.
  • S3 write→read(match)→delete→gone through S3ArtifactStore on hasna-oss-knowledge-prod-789877399345.
  • bun run build green (tsc); bun test cli+cloud-storage-kit+artifact-store = 59 pass / 0 fail.

Not in scope

  • No publish. Machine default stays local; flipping default is an env/fleet-config action (documented), not a code change.

Add a knowledge_items Postgres table and a cloud-catalog access layer so the
CLI catalog (add/list/get) round-trips to the cloud DB in cloud storage mode,
instead of the local JSON store. Reads/writes go through the vendored storage
kit (createKnowledgeCloudClient). Remaining catalog mutators are explicitly
guarded in cloud mode so they never silently touch the local JSON store.

- src/db/pg-migrations.ts: knowledge_items table + indexes (additive)
- src/db/cloud-catalog.ts: list/get/count/upsert/add via storage kit
- src/cli.ts: route add/list/get to cloud when HASNA_KNOWLEDGE_STORAGE_MODE=cloud;
  guard update/archive/restore/untag/delete/upsert/prune/dedupe/export/stats/inventory
- rebuilt bin/ + dist/
Idempotent upsert-by-id importer for KnowledgeItem JSONL exports into the cloud
knowledge_items table, mirroring apply-cloud-migrations.mjs (owner DSN, no echo).
Documents schema apply, catalog import, real-code-path + S3 verification, local
store backup, and the default-mode flip (env/fleet-config, reversible, no code
release). Records the tunnel sslmode=require->uselibpqcompat gotcha.
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

OpenLoops worker review: blocked, not safe to merge yet.

Evidence gathered on 2026-07-06:

  • GitHub merge gate is red: mergeable=CONFLICTING, mergeStateStatus=DIRTY.
  • Branch is stale/divergent from current main: git rev-list --left-right --count origin/main...origin/oss-cloud-runtime/knowledge-cloud-catalog-cutover => 1 3.
  • Dry merge reports generated-artifact conflicts, especially bin/knowledge-mcp.js, bin/knowledge.js, and dist/index.js; the generated bundle conflict includes conflict markers in bin/knowledge.js.
  • CI has one failed check: test-matrix (windows-latest, bun) from run 28788646710; other listed Ubuntu/macOS checks passed.
  • PR head still reports package version 0.2.77, while the repo graph has a later release/knowledge-0.2.78 branch/commit, so release/version ordering needs reconciliation before merge.
  • Local validation in the isolated worktree was inconclusive, not green: bun run build failed before compile due missing dependencies (commander, @hasna/events/commander), and bun test finished 112 pass / 2 skip / 21 fail / 7 errors with missing packages, Bun tempdir write errors in spawned CLI subprocesses, and several 5s timeouts.

Adversarial review also flagged non-blocking-but-important follow-ups before merge:

  • Add focused coverage for src/db/cloud-catalog.ts, CLI cloud routing, and scripts/import-cloud-catalog.mjs; current tests cover storage-kit basics but not this catalog path directly.
  • Recheck short_id semantics: knowledge_items.short_id is indexed but not unique, while cloudGetItem() uses WHERE id = $1 OR short_id = $1 LIMIT 1, which can return arbitrary rows if duplicates exist.
  • Consider pushing search/filter/pagination into SQL or explicitly accepting the current all-row cloudListItems() behavior for catalog size.

Required merge gate: rebase/merge current origin/main, resolve generated artifact conflicts, regenerate/verify artifacts (bun run verify:generated), rerun CI including Windows, and resolve the release/version story. I did not merge.

@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Closing as superseded / already-fixed-on-main.

Since this PR was opened (2026-07-06), main was reconciled to the published npm line (0.2.86 → now 0.2.87, PR #31/#20/#19/#16 etc.), which re-implements the exact functionality this PR proposed, but via a different (API-mediated) architecture:

  • knowledge_items table + indexes — already on main at src/db/pg-migrations.ts:344 (identical additive migration).
  • PURE REMOTE / A1 cloud-catalog CRUD — already on main in src/serve.ts:80-244 (list/get/count/upsert/add/update/delete on knowledge_items via createKnowledgeCloudClient, comment: 'knowledge_items in the cloud Postgres (PURE REMOTE / A1)'). This PR's standalone src/db/cloud-catalog.ts duplicates that.
  • createKnowledgeCloudClient — already on main at src/db/remote-storage.ts:25.
  • cloud storage-mode wiring — already on main (src/db/storage-sync.ts StorageMode='local'|'cloud' with self_hosted→cloud alias; src/cloud-store.ts routes item CRUD to the cloud API).

The PR is also CONFLICTING/DIRTY against main and bundles stale build artifacts (bin/knowledge.js, bin/knowledge-mcp.js, dist/index.js, dist/storage.js) built from the pre-reconcile July-6 base, which would clobber the current 0.2.87 builds and regress the fleet. The one-shot 48-record catalog import it describes was already performed. Nothing merge-worthy remains after re-scoping. No publish, no code change needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant