Skip to content

chore(selfhost): unify queue/adapter/vectorize backend contracts - #4188

Merged
JSONbored merged 1 commit into
mainfrom
chore/selfhost-shared-backend-contract-4010
Jul 8, 2026
Merged

chore(selfhost): unify queue/adapter/vectorize backend contracts#4188
JSONbored merged 1 commit into
mainfrom
chore/selfhost-shared-backend-contract-4010

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Adds src/selfhost/backend-contracts.ts: one shared, project-owned interface per self-host swappable-backend pair (queue, D1-shaped storage adapter, Vectorize-shaped RAG store), replacing the sqlite/postgres queue pair's two independently-declared interfaces (DurableQueue/PgDurableQueue) and the T | Promise<T> union server.ts used to paper over their drift, and inserting a real typed checkpoint before each adapter's unavoidable as unknown as <ambient type> cast (D1Database/Vectorize are declare abstract class, so that cast itself can't be removed — but nothing previously checked the object's own shape against it, or against its sibling, first).
  • Makes the sqlite queue backend's introspection/admin surface (size, deadCount, processingCount, stats, snapshot, pressureSignals, the dead-letter admin methods, topBacklogRepos, releaseStaleForegroundDeferrals) fully async — a strict superset of its former sync surface, matching the Postgres backend exactly — and adds a documented no-op init(). Fixed a latent bug this surfaced: releaseStaleForegroundDeferralsSafely's try/catch needed to become await-based to keep catching errors from the now-async call (there's an existing regression test for exactly this failure mode).
  • Fixes the returnMetadata divergence: only vectorize.ts (sqlite)'s private QueryOptions declared it. Traced the real call path (src/review/adapters.ts's reviewVectorAdaptervectorize.query(vector, opts), fed by src/review/rag.ts, which always sets returnMetadata: "all") and confirmed every backend is invoked with this option identically regardless of which is bound to env.VECTORIZE — so it belongs on all three, typed as the real three-value union Cloudflare's own VectorizeQueryOptions.returnMetadata uses, not removed from vectorize.ts.
  • Adds a contract test per pair under test/contract/ (describe.each / shared-spec-function style) that runs the identical assertion suite against every concrete implementation of each pair, alongside — not replacing — each backend's own existing implementation-specific test file. Also adds test/unit/selfhost-pg-adapter.test.ts (mocked pg.Pool, no real Postgres needed), closing a gap where pg-adapter.ts's own D1 surface (batch/first/run/raw) was previously only exercised by the real-Postgres integration test.

Scope

Validation

  • git diff --check
  • npm run actionlint (unaffected by this diff; no workflow files touched)
  • npm run typecheck
  • npm run test:coverage locally — full unsharded run, 593 passed / 2 skipped files, 12073 passed / 12 skipped tests; targeted per-file coverage checks confirm every line this PR touches in a non-Codecov-ignored file (sqlite-queue.ts, d1-adapter.ts, vectorize.ts, qdrant-vectorize.ts, pg-vectorize.ts) is covered (pg-queue.ts/pg-adapter.ts/server.ts are Codecov-ignored per codecov.yml, validated by the real-Postgres integration test instead)
  • npm run test:workers
  • npm run build:mcp (unaffected; not touched, but ran clean)
  • npm run test:mcp-pack
  • npm run ui:openapi:check (unaffected — no API/schema changes)
  • npm run ui:lint (unaffected — no apps/gittensory-ui changes)
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate — 0 vulnerabilities
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries — see the contract tests and the new selfhost-pg-adapter.test.ts above

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics. (n/a — no PR-facing text generated by this code)
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (n/a — no auth/CORS/session code touched)
  • API/OpenAPI/MCP behavior is updated and tested where needed. (n/a — no API shape changes)
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (n/a — backend-only change)
  • Visible UI changes include a UI Evidence section. (n/a — no UI change)
  • Public docs/changelogs are updated where needed. (n/a — no changelog edit in this PR)

UI Evidence

Not applicable — this is a backend-only change (self-host queue/adapter/vectorize contracts + tests); no visible UI surface changed.

Notes

  • Traced ≥2 existing analogues before designing the new interfaces: src/selfhost/queue-common.ts's existing SelfHostQueueDeadLetterAdmin/DeadLetterJob pattern (a project-defined interface shared by both queue backends), src/selfhost/d1-adapter.ts's existing SqliteDriver (a minimal project-defined interface abstracting a concrete dependency, already imported by three other self-host files), and src/review/rag.ts's VectorAdapter/VectorMatch (a structurally-similar but distinct higher-layer interface that wraps whichever Vectorize-shaped binding is active).
  • Did not touch src/queue/processors.ts, the visual-capture epic, or anything under the config-as-code epic (Epic: config-as-code should fully control every review feature, not just restrict it #4092).

Defines one shared, project-owned interface per self-host swappable-backend
pair in a new src/selfhost/backend-contracts.ts (#4010):

- DurableQueue replaces the sqlite/postgres queue backends' two
  independently-declared interfaces (DurableQueue/PgDurableQueue), which had
  drifted (every sqlite method sync vs the postgres equivalent Promise-
  wrapped, plus an init() only postgres had) and were reconciled only by a
  loose T | Promise<T> union on server.ts's Backend.queue. Every method is
  now Promise-returning on both backends; the sqlite side's init() is a
  documented no-op since its setup already runs synchronously at construction.
- SelfHostD1Database/SelfHostD1PreparedStatement give d1-adapter.ts and
  pg-adapter.ts a real typed checkpoint before their previously-blind
  `as unknown as D1Database` cast (D1Database is a `declare abstract class`,
  so that final cast is unavoidable, but nothing previously checked either
  adapter's own shape against it or its sibling first).
- SelfHostVectorRecord/SelfHostVectorizeQueryOptions/SelfHostVectorizeMatch/
  SelfHostVectorize do the same for vectorize.ts, qdrant-vectorize.ts, and
  pg-vectorize.ts, and fix the one already-manifested divergence: only
  vectorize.ts's QueryOptions carried returnMetadata. Traced the real call
  path (src/review/adapters.ts's reviewVectorAdapter, fed by src/review/
  rag.ts, which always sets returnMetadata: "all") and confirmed every
  backend is invoked with this option identically regardless of which is
  bound to env.VECTORIZE, so it belongs on all three - typed as the real
  three-value union Cloudflare's own VectorizeQueryOptions.returnMetadata
  uses, not removed from vectorize.ts.

Adds a contract test per pair under test/contract/ that runs the same
assertion suite against every concrete implementation of each pair, and a
new selfhost-pg-adapter.test.ts covering pg-adapter's own D1 surface with a
mocked pg.Pool (previously only exercised by the real-Postgres integration
test). Both alongside, not replacing, each backend's existing
implementation-specific test file.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.74%. Comparing base (f46aa4a) to head (847a5de).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4188      +/-   ##
==========================================
- Coverage   93.74%   93.74%   -0.01%     
==========================================
  Files         387      387              
  Lines       36398    36397       -1     
  Branches    13330    13330              
==========================================
- Hits        34120    34119       -1     
  Misses       1621     1621              
  Partials      657      657              
Files with missing lines Coverage Δ
src/selfhost/d1-adapter.ts 100.00% <100.00%> (ø)
src/selfhost/pg-vectorize.ts 100.00% <100.00%> (ø)
src/selfhost/qdrant-vectorize.ts 100.00% <100.00%> (ø)
src/selfhost/sqlite-queue.ts 99.60% <100.00%> (ø)
src/selfhost/vectorize.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 8, 2026
@JSONbored JSONbored self-assigned this Jul 8, 2026
@loopover-orb

loopover-orb Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 Gittensory is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@JSONbored
JSONbored merged commit dc711b9 into main Jul 8, 2026
11 checks passed
@JSONbored
JSONbored deleted the chore/selfhost-shared-backend-contract-4010 branch July 8, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Development

Successfully merging this pull request may close these issues.

chore(selfhost): shared contract + tests for queue/adapter/vectorize backend pairs

1 participant