Migrate reads off published_realms#4581
Merged
lukemelia merged 4 commits intoMay 4, 2026
Merged
Conversation
Contributor
Host Test Results 1 files ± 0 1 suites ±0 1h 11m 36s ⏱️ - 1h 30m 4s For more details on these failures and errors, see this check. Results for commit 2fe8200. ± Comparison against base commit f22c69b. |
Contributor
Realm Server Test Results 1 files ±0 1 suites ±0 16m 5s ⏱️ - 1m 15s For more details on these failures, see this check. Results for commit 2fe8200. ± Comparison against base commit f22c69b. |
lukemelia
added a commit
that referenced
this pull request
Apr 30, 2026
Brings origin/main onto the branch, which most importantly carries 4758677 ("Regenerate SQLite schema after the new migration (CS-10962)"). The new schema file includes realm_registry, which the host/SQLite test environment was missing — the cause of the host shard failures observed on PR #4581 (every shard hit "no such table: realm_registry" because Phase 4 reads now go through that table). Conflict resolution in packages/realm-server/server.ts:createRealm: kept HEAD's withRealmWriteLock + mirrorSourceRealmToRegistry control flow and adopted main's split of the on-disk metadata files (.realm.json now holds only {publishable: true}; the new realm.json holds the RealmConfig card with cardInfo/iconURL/backgroundURL). The loadRealms()/findPublishedRealms() block from main was dropped — HEAD already replaced that boot path with the reconciler-driven lazy mount. Tests that were inserting directly into published_realms but bypassing the dual-write are migrated to also write realm_registry, since reads have now moved off published_realms: - tests/helpers/index.ts: published-realm fixture now also calls mirrorPublishedRealmToRegistry. Fixes the index-responses-test.ts "Published realm index responses" group (~6 tests on shard 2/6) and any other consumer of setupPermissionedRealm({published:true}). - tests/queries-test.ts: both insertPublishedRealm helpers (fetchUserPermissions module + fetchAllRealmsWithOwners module) now dual-write. Fixes 4 failures on shard 6/6. - tests/full-reindex-test.ts: insertPublishedRealm helper dual-writes. Fixes "enqueues jobs for source and published realms" on shard 3/6 (fullReindex calls fetchAllRealmsWithOwners which reads realm_registry). - tests/server-endpoints/index-responses-test.ts: the "ETag changes after republishing" test bumps last_published_at on both tables now (the ETag derives from the realm_registry value). Tests that intentionally read published_realms directly are left unchanged: realm-registry-backfill-test.ts is testing the backfill; delete-realm-test.ts SELECTs verify the legacy dual-write path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Observability diff (vs staging)No dashboard / folder changes detected against the staging Grafana. (Run: https://github.com/cardstack/boxel/actions/runs/25329561095) |
Contributor
Preview deploymentsHost Test Results 1 files 1 suites 2h 0m 53s ⏱️ Results for commit bb688de. Realm Server Test Results 1 files ± 0 1 suites +1 18m 10s ⏱️ + 18m 10s Results for commit bb688de. ± Comparison against earlier commit 632353f. |
lukemelia
marked this pull request as ready for review
May 1, 2026 02:28
Contributor
There was a problem hiding this comment.
Pull request overview
This PR continues the DB-Authoritative Realm Registry migration by switching runtime reads from published_realms to realm_registry (gated by kind='published'), and also includes several adjacent changes spanning realm config storage, CLI output control, harness stability, error persistence safety, and observability tooling.
Changes:
- Migrate runtime SQL reads from
published_realmstorealm_registryacross realm-server and runtime-common query paths. - Migrate realm config “card-owned” fields into a
RealmConfigcard instance at/realm.json, adjusting parsing, PATCH behavior, fixtures, and tests accordingly. - Improve tooling/ops: add
boxelglobal--quiet, add Software Factory port reservation/holding to avoid EADDRINUSE races, clamp oversizederror_docbefore DB persistence, and extend observability scripts/workflows (Grafana/Loki + PR status comments).
Reviewed changes
Copilot reviewed 116 out of 117 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/claude-aws.sh | New helper script to assume a dedicated readonly AWS role for Claude workflows via MFA and write temp creds to an AWS profile. |
| mise-tasks/claude-aws | Mise task wrapper for scripts/claude-aws.sh. |
| pnpm-workspace.yaml | Bumps ember-concurrency catalog version (major upgrade). |
| pnpm-lock.yaml | Lockfile updates for ember-concurrency@5.2.0 and dependents. |
| packages/base/realm-config.gts | Introduces RealmConfig card definition and routing rule field. |
| packages/base/realm.json | Adds Base realm’s realm.json card instance. |
| packages/base/.realm.json | Removes card-owned fields from legacy sidecar. |
| packages/base/links-to-many-component.gts | Updates task typing/usage for new ember-concurrency API. |
| packages/base/links-to-editor.gts | Updates task typing/usage and minor template formatting. |
| packages/runtime-common/realm.ts | Adds realm registry reads for published realm metadata; adds realm.json overlay to realmInfo; splits PATCH /_config writes between realm.json and .realm.json; updates cache invalidation. |
| packages/runtime-common/db-queries/realm-permission-queries.ts | Replaces published realm subqueries with realm_registry(kind='published') equivalents; preserves caller field names via SQL aliases. |
| packages/realm-server/server.ts | Uses realm_registry for connect-origin and lastPublishedAt lookups; seeds new realms with both .realm.json (legacy) and realm.json (card). |
| packages/realm-server/utils/realm-readability.ts | Switches getPublishedRealmURLs to read from realm_registry. |
| packages/realm-server/handlers/handle-publish-realm.ts | Existence/identity checks now read from realm_registry; adds a second full-index pass with clearLastModified for prerender correctness. |
| packages/realm-server/handlers/handle-unpublish-realm.ts | Full-row read now comes from realm_registry with aliases matching legacy accessors. |
| packages/realm-server/handlers/handle-delete-realm.ts | Published-existence and cascade lookups now use realm_registry (kind='published'). |
| packages/realm-server/handlers/handle-download-realm.ts | Published realm path resolution now uses realm_registry.disk_id (aliased to legacy id). |
| packages/runtime-common/error.ts | Adds clampSerializedError and budget constants to prevent oversized jsonb error docs. |
| packages/runtime-common/index-writer.ts | Clamps serialized error docs before persistence to DB. |
| packages/runtime-common/definition-lookup.ts | Clamps module error docs before storing in DB. |
| packages/runtime-common/index.ts | Re-exports clamp/budget constants from runtime-common. |
| packages/realm-server/tests/clamp-serialized-error-test.ts | Adds QUnit coverage for error clamping behavior. |
| packages/realm-server/tests/index.ts | Registers new clamp test file. |
| packages/host/tests/unit/index-writer-test.ts | Adds host-side persistence tests validating in-budget passthrough and over-budget clamping. |
| packages/realm-server/tests/server-endpoints/index-responses-test.ts | Updates republish simulation to update both legacy table and realm_registry. |
| packages/realm-server/tests/queries-test.ts | Mirrors published realm rows into realm_registry for tests. |
| packages/realm-server/tests/helpers/index.ts | Mirrors published realm fixtures into realm_registry for tests. |
| packages/realm-server/tests/full-reindex-test.ts | Mirrors published realm fixtures into realm_registry for tests. |
| packages/realm-server/tests/server-endpoints/realm-lifecycle-test.ts | Updates assertions for realm.json vs .realm.json ownership split. |
| packages/realm-server/tests/server-endpoints/maintenance-endpoints-test.ts | Updates indexing counts due to new realm.json being indexed. |
| packages/realm-server/tests/publish-unpublish-realm-test.ts | Updates expected realm version due to extra full-index pass on publish. |
| packages/realm-server/tests/types-endpoint-test.ts | Adds expected summary entry/icon for RealmConfig card type. |
| packages/realm-server/tests/realm-endpoints-test.ts | Updates config PATCH tests for card-owned fields; ensures realm.json is present in index resources. |
| packages/realm-server/tests/cards/realm.json | Adds test fixture realm.json card. |
| packages/realm-server/tests/cards/.realm.json | Clears legacy fixture sidecar. |
| packages/realm-server/scripts/setup-submission-realm.sh | Creates submission realm.json card while keeping legacy sidecar for remaining flags. |
| packages/realm-server/scripts/migrate-realm-config-to-card.sh | Adds migration script to create realm.json from .realm.json and trim migrated fields. |
| packages/software-factory/src/harness/shared.ts | Adds findAndHoldAvailablePort + updates SOURCE_REALM_GLOB to include realm.json. |
| packages/software-factory/src/harness/support-services.ts | Uses port reservations to prevent port allocation races before spawning services. |
| packages/software-factory/src/harness/database.ts | Uses port reservations for worker-manager + adds fetch timeout for indexing status polling. |
| packages/software-factory/tests/find-and-hold-available-port.test.ts | Adds tests for holder semantics and concurrency safety. |
| packages/software-factory/tests/index.ts | Registers new port-holder test. |
| packages/software-factory/tests/helpers/test-client.ts | Enables boxel-cli quiet mode during tests to reduce CI log noise; resets on cleanup. |
| packages/software-factory/src/factory-tool-executor.ts | Adds debug flag to config; composes boxel CLI invocations with --quiet by default. |
| packages/software-factory/src/factory-issue-loop-wiring.ts | Wires factory --debug into ToolExecutor config. |
| packages/software-factory/tests/factory-tool-executor.test.ts | Adds unit tests for composeBoxelCliInvocation. |
| packages/software-factory/realm/realm.json | Adds Software Factory realm.json card instance. |
| packages/software-factory/realm/.realm.json | Clears legacy sidecar fields. |
| packages/software-factory/test-fixtures/test-realm-runner/realm.json | Adds fixture realm.json card instance. |
| packages/software-factory/test-fixtures/test-realm-runner/.realm.json | Clears legacy sidecar fields. |
| packages/software-factory/test-fixtures/darkfactory-adopter/realm.json | Adds fixture realm.json card instance. |
| packages/software-factory/test-fixtures/darkfactory-adopter/.realm.json | Clears legacy sidecar fields. |
| packages/software-factory/test-fixtures/bootstrap-target/realm.json | Adds fixture realm.json card instance. |
| packages/software-factory/test-fixtures/bootstrap-target/.realm.json | Clears legacy sidecar fields. |
| packages/catalog-realm/realm.json | Adds catalog realm.json card instance. |
| packages/catalog-realm/.realm.json | Clears legacy sidecar fields. |
| packages/experiments-realm/realm.json | Adds experiments realm.json card instance. |
| packages/experiments-realm/.realm.json | Clears legacy sidecar fields. |
| packages/openrouter-realm/realm.json | Adds openrouter realm.json card instance. |
| packages/openrouter-realm/.realm.json | Clears legacy sidecar fields. |
| packages/host/app/components/operator-mode/workspace-chooser/workspace.gts | Excludes realm.json from workspace content summaries alongside .realm.json. |
| packages/host/tests/cards/realm.json | Adds host test realm.json fixture. |
| packages/host/tests/cards/.realm.json | Clears legacy sidecar fields in host test fixture. |
| packages/host/memory-baseline.json | Updates memory baseline snapshot values/date. |
| packages/host/config/schema/1777413435523_schema.sql | Adds timing_diagnostics column to schema snapshot(s). |
| packages/host/config/schema/1776964391615_schema.sql | Removes older schema snapshot file. |
| packages/boxel-cli/src/lib/cli-log.ts | Adds centralized CLI logging + console interception for --quiet. |
| packages/boxel-cli/src/index.ts | Adds global --quiet option and early quiet-mode activation. |
| packages/boxel-cli/api.ts | Exposes setQuiet/isQuiet for in-process consumers (tests/SF). |
| packages/boxel-cli/src/lib/auth.ts | Adds getUserRealmsFromMatrixAccountData helper; refactors URL building. |
| packages/boxel-cli/src/lib/profile-manager.ts | Adds getUserRealms() method. |
| packages/boxel-cli/src/commands/search.ts | Uses cliLog.output for JSON/result payloads. |
| packages/boxel-cli/src/commands/run-command.ts | Uses cliLog.output for JSON/result payloads. |
| packages/boxel-cli/src/commands/read-transpiled.ts | Uses cliLog.output for JSON and raw content output. |
| packages/boxel-cli/src/commands/file/read.ts | Uses cliLog.output for JSON and raw content output. |
| packages/boxel-cli/src/commands/file/write.ts | Uses cliLog.output for JSON payload output. |
| packages/boxel-cli/src/commands/file/list.ts | Uses cliLog.output for JSON payload output. |
| packages/boxel-cli/src/commands/file/lint.ts | Uses cliLog.output for JSON payload output. |
| packages/boxel-cli/src/commands/file/delete.ts | Uses cliLog.output for JSON payload output. |
| packages/boxel-cli/src/commands/realm/list.ts | Adds new boxel realm list command (accessible vs hidden realms via Matrix account data). |
| packages/boxel-cli/src/commands/realm/index.ts | Registers new realm list command. |
| packages/boxel-cli/src/commands/realm/cancel-indexing.ts | Uses cliLog.output for JSON payload output. |
| packages/boxel-cli/tests/lib/cli-log.test.ts | Adds Vitest unit tests for quiet mode + cliLog routing. |
| packages/boxel-cli/tests/smoke.test.ts | Adds e2e checks for --quiet help + behavior. |
| packages/boxel-cli/tests/integration/realm-list.test.ts | Adds integration coverage for realm list behavior. |
| packages/postgres/package.json | Changes migrate script options (no-check-order, verbose=false). |
| packages/postgres/migrations/1777413435523_setup-claude-readonly-db-user.js | Adds migration to provision a Claude-specific readonly DB user in staging/production. |
| packages/observability/scripts/tail-logs.sh | Adds Loki tail script (auth + LogQL + polling) for local/staging/prod. |
| packages/observability/scripts/apply-datasources.sh | Adds hosted Grafana datasource upsert via HTTP API from provisioning YAMLs. |
| packages/observability/scripts/apply.sh | Adds preflight for hosted deps/env vars; invokes datasource apply after grafanactl push. |
| packages/observability/scripts/diff.sh | Normalizes pulled grafanactl tree to committed layout for meaningful diffs. |
| packages/observability/provisioning/datasources/boxel-db.yaml | Adds Postgres datasource provisioning YAML using envsubst placeholders. |
| packages/observability/provisioning/datasources/boxel-cloudwatch.yaml | Adds CloudWatch datasource provisioning YAML. |
| packages/observability/provisioning/datasources/synapse-prometheus.yaml | Adds AMP Prometheus datasource provisioning YAML. |
| packages/observability/provisioning/datasources/boxel-db.json | Removes legacy JSON datasource manifest. |
| packages/observability/provisioning/datasources/boxel-cloudwatch.json | Removes legacy JSON datasource manifest. |
| packages/observability/provisioning/datasources/synapse-prometheus.json | Removes legacy JSON datasource manifest. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/boxel-jobs.json | Adds folder annotation metadata for dashboard. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/boxel-logs.json | Adds folder annotation metadata for dashboard. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/realm-permissions.json | Adds folder annotation metadata for dashboard. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/synapse.json | Adds folder annotation metadata for dashboard. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/user-credits.json | Adds folder annotation metadata for dashboard. |
| packages/observability/grafanactl/resources/dashboards/boxel-status/worker-status.json | Adds folder annotation metadata for dashboard. |
| packages/observability/README.md | Documents new datasource apply flow and tail-logs usage/label schema updates. |
| .claude/skills/tail-logs/SKILL.md | Adds Claude skill wrapper docs for tail-logs. |
| .github/workflows/preview-host.yml | Switches preview commenting to slot-based PR status comment workflow. |
| .github/workflows/pr-boxel-ui.yml | Switches preview commenting to slot-based PR status comment workflow. |
| .github/workflows/preview-comment.yml | Changes from posting sticky PR comment to uploading per-slot artifacts + outputting slot list. |
| .github/workflows/pr-status-comment.yml | New reusable workflow to assemble/upsert a unified sticky PR status comment from slot artifacts. |
| .github/workflows/ci.yaml | Uploads realm test summary as a pr-comment slot artifact and triggers PR status update. |
| .github/workflows/ci-host.yaml | Uploads host test summary as a pr-comment slot artifact and triggers PR status update. |
| .github/workflows/observability-apply-staging.yml | Expands SSM fetch to include datasource placeholders (Loki/DB/Prometheus) for apply. |
| .github/workflows/observability-apply-production.yml | Expands SSM fetch to include datasource placeholders (Loki/DB/Prometheus) for apply. |
| packages/matrix/tests/create-realm.spec.ts | Updates expected UI element count (likely due to realm config/index changes). |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
backspace
approved these changes
May 1, 2026
Phase 4 PR 1. After this commit, every runtime read of published_realms in production code paths goes through realm_registry instead. The legacy table is still dual-written from mutation handlers — Phase 4 PR 2 drops those writes and the table itself. Sites migrated: - server.ts:395 (POST /connect origin lookup) - server.ts:673 (getPublishedRealmInfo lastPublishedAt) - handle-publish-realm.ts (existence check + identity lookup before the dual-written INSERT/UPDATE; switched from .id to .disk_id since realm_registry's primary key is a separate uuid from the published realm's disk uuid) - handle-unpublish-realm.ts (full row read for response + permission + path compose; SQL aliases preserve the legacy field names so the downstream accessors are unchanged) - handle-delete-realm.ts (existence check rejecting deletes on published URLs + cascade lookup of published copies sourced from this realm; same SQL-alias pattern) - handle-download-realm.ts (path resolution for published realms) - utils/realm-readability.ts (getPublishedRealmURLs membership probe) - runtime-common/db-queries/realm-permission-queries.ts (3 sites: the "exclude published realm URLs" subqueries used by fetchUserPermissions and the published-owner fallback) - runtime-common/realm.ts (queryPublishedRealm + querySourceRealmPublications) Each migrated query gates on `kind = 'published'` and reads realm_registry's url / source_url / disk_id / last_published_at / owner_username columns. Where legacy callers used field names from published_realms (id, source_realm_url, published_realm_url), SQL column aliases keep the in-process accessors stable so this PR is a pure data-source swap with no caller-side renames. Test files (delete-realm-test.ts) keep their direct published_realms reads — they're verifying the dual-write side that this PR doesn't touch. realm-registry-backfill.ts also keeps its read; it's the backfill source by design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…robe
Copilot flagged the `?column?` row-shape assertion: that key is
Postgres's default unnamed-column label, which isn't portable across
SQL adapters and is easy to break under any future schema-test or
adapter swap. Alias the literal to `AS found` and type the row as
`{ found: number }[]` to make the contract explicit.
The check itself only looks at `.length`, so the alias is purely a
hygiene change — no behavior delta.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…path
These test fixtures insert directly into published_realms but were
bypassing the dual-write into realm_registry. Phase 4 PR 1 moved
the runtime reads to realm_registry, so any fixture-only published
realms must also write the registry row to be visible.
- tests/helpers/index.ts: published-realm fixture now also calls
mirrorPublishedRealmToRegistry. Fixes the index-responses-test.ts
"Published realm index responses" group and any other consumer of
setupPermissionedRealm({published:true}).
- tests/full-reindex-test.ts: insertPublishedRealm helper dual-writes
(fullReindex calls fetchAllRealmsWithOwners which reads
realm_registry).
- tests/server-endpoints/index-responses-test.ts: the "ETag changes
after republishing" test bumps last_published_at on both tables now
(the ETag derives from the realm_registry value).
Tests that intentionally read published_realms directly are left
unchanged: realm-registry-backfill-test.ts is testing the backfill;
delete-realm-test.ts SELECTs verify the legacy dual-write path.
(Restored from the merge commit 0014e07 body during rebase onto
main; the parent migration commit is unchanged.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lukemelia
force-pushed
the
cs-10896-phase-4-pr-1-migrate-reads-off-published-realms
branch
from
May 4, 2026 16:09
a2d4645 to
6c47a7e
Compare
lukemelia
changed the base branch from
cs-10895-phase-3-pr-2-remove-direct-mount-from-handlers
to
main
May 4, 2026 16:09
Contributor
Author
|
Rebased onto Branch went from 8 commits → 3 commits: Kept
Dropped (superseded by what landed on main)
|
Same pattern as the other test fixtures that insert directly into published_realms but bypass the dual-write into realm_registry. Phase 4 PR 1 moved fetchUserPermissions and fetchAllRealmsWithOwners to read from realm_registry, so the four affected tests (queries-test.ts:54, :94, :184, :212) need the fixture to dual-write or the published rows are invisible. Mirrors the changes already added to tests/helpers/index.ts, full-reindex-test.ts, and index-responses-test.ts in 6c47a7e. The merge-commit body that originally added those evidently missed queries-test.ts; CI on the rebased branch surfaced the gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lukemelia
force-pushed
the
cs-10896-phase-4-pr-1-migrate-reads-off-published-realms
branch
from
May 4, 2026 17:07
632353f to
bb688de
Compare
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 PR 1 of the DB-Authoritative Realm Registry project. Switches every runtime read of `published_realms` to `realm_registry` (gated on `kind = 'published'`). The legacy table is still dual-written from mutation handlers — Phase 4 PR 2 drops those writes and the table itself.
Stacked on #4561 (CS-10895). Merge that first.
Sites migrated
Enumerated in the spec:
Additional sites found via grep:
Approach
Each migrated query gates on `kind = 'published'` and reads from `realm_registry`'s columns (`url` / `source_url` / `disk_id` / `last_published_at` / `owner_username`). Where the legacy caller used `published_realms` field names (`id`, `source_realm_url`, `published_realm_url`), SQL column aliases preserve the in-process accessor names — so this PR is a pure data-source swap with no caller-side renames.
The one place where the field name changes is `handle-publish-realm.ts` existence check, which now uses `existingRows[0].disk_id` instead of `existingRows[0].id` (`realm_registry.id` is a separate uuid from `published_realms.id` — the latter matches `realm_registry.disk_id` for kind='published' rows by design, see migration `1776960113000`).
Out of scope
Test plan
🤖 Generated with Claude Code