diff --git a/src/selfhost/docs-drift-audit-sweep-rfc.md b/src/selfhost/docs-drift-audit-sweep-rfc.md new file mode 100644 index 0000000000..dcb4e99eb8 --- /dev/null +++ b/src/selfhost/docs-drift-audit-sweep-rfc.md @@ -0,0 +1,116 @@ +# Scheduled docs-drift audit sweep — design doc (#3048) + +Part of #1819 (self-host docs accuracy audit). Siblings: the mechanical presence/shape drift lint +(`scripts/check-docs-drift.mjs`, "Phase 1") and schema-anchored examples ("Phase 2"). This RFC covers +the catch-all Phase 3: a recurring, low-noise sweep that catches **behavioral/prose** drift — a docs +claim that was true when written and quietly stopped being true — which neither sibling can catch +mechanically. Per the issue's own deliverables, this doc is the first step; the actual scheduled job +lands once the design below is agreed. + +## What the two mechanical siblings already catch (and don't) + +`scripts/check-docs-drift.mjs` cross-checks five enumerable **presence** surfaces against specific docs +pages/examples: `LOOPOVER_REVIEW_*` feature flags (`src/env.d.ts`), `@loopover` command catalogs +(`src/github/commands.ts`), `*GateMode` fields (`src/types.ts`'s `RepositorySettings`), and the full +`RepositorySettings` + `FocusManifest` field surface against `.loopover.yml.example` +(`check-docs-drift.mjs:1-11`). It answers "does every flag/command/field that exists in code also appear +somewhere in docs?" — a shape/presence check, anchored on declaration syntax (`extractLoopOverReviewFlags`, +`extractGateModeFields`, etc.), not on whether the *prose describing* that surface is still accurate. + +That is precisely the gap this issue targets. The reported motivating case: "blocking is always +confirmed-contributor-gated" was duplicated verbatim across `docs.github-app.tsx`, +`docs.how-reviews-work.tsx`, and `docs.tuning.tsx`, and was wrong — the gate's confirmed-contributor +handling had changed underneath the claim. Every surface the claim touches (a `*GateMode` field, an env +flag) still existed and was still documented *somewhere*; the mechanical lint had nothing to flag, +because presence isn't behavior. + +## Proposal: reuse the drift-report → consolidated-issue pipeline `fileUpstreamDriftIssues` already runs + +`src/upstream/ruleset.ts`'s `fileUpstreamDriftIssues` (`ruleset.ts:275-331`) is an existing, working, +in-production instance of exactly the shape this sweep needs, already satisfying every hard requirement +below: + +- **Kill-switch respecting**: checks `isGlobalAgentPause`/`isGlobalAgentFrozen` before any write + (`ruleset.ts:283-285`), and the whole mechanism is flag-gated off by default + (`LOOPOVER_AUTO_FILE_DRIFT_ISSUES`, `ruleset.ts:276-278`). +- **One issue per finding-run, not one per finding**: each open drift *report* maps to exactly one + GitHub issue, found by a stable content fingerprint (`findGitHubIssueForFingerprint`) and then + **updated in place** on a later run rather than re-created (`ruleset.ts:296-316`), with a no-op-diff + short-circuit so an unresolved-but-unchanged report never spuriously re-notifies + (`driftIssueUnchanged`, `ruleset.ts:302-309`). +- **Audited**: emits a single `upstream.drift_issues_filed` audit event per run with created/updated/ + skipped/unchanged counts (`ruleset.ts:326-330`), the same pattern `audit_events` already uses + pervasively (`src/queue/processors.ts`). +- **Cron-wired at a deliberately slow cadence**: `refresh-upstream-drift` runs on the hourly tick + (`src/index.ts:225`), but `file-upstream-drift-issues` itself only fires inside the six-hourly + **full-sync window** (`src/index.ts:270-275`) — i.e. the detection and the issue-filing steps are + already split into two differently-paced jobs, which is the same split this proposal needs (see + below). + +The proposal is to add a **new, analogous pair** — `docs-drift-audit-sweep` (detection: reads N docs +pages, re-verifies their behavioral claims, writes drift reports) and `file-docs-drift-issues` +(filing: turns open reports into one consolidated issue, reusing `fileUpstreamDriftIssues`'s +fingerprint/update-not-duplicate logic near-verbatim) — rather than inventing a new +detect-and-file-in-one-step mechanism. This is "reuse the most existing infrastructure" read literally: +the *filing* half needs no new design, only a second table + a second `env.LOOPOVER_AUTO_FILE_*` flag +mirroring the existing one. + +## What's actually new: the detection step + +Nothing in this codebase today reads a docs page's prose and re-verifies a specific behavioral claim +against current source — that half genuinely needs new code, gated behind the self-host AI provider +abstraction (`src/selfhost/ai.ts`'s `SelfHostAi`, the same interface `aiReviewMode`/`slopAiAdvisory` +already run their AI passes through, e.g. `src/services/ai-review.ts`). Concretely: + +1. **Input**: one docs route file (e.g. `apps/loopover-ui/src/routes/docs.tuning.tsx`) plus a + maintainer-curated list of the `src/`/`packages/` files it makes behavioral claims about. The 2026-07-04 + manual pass already established this page↔source mapping is tractable — the existing + `docs.self-hosting-docs-audit.tsx` page (`SELFHOST_DOCS_PAGES` / + `SELFHOST_SOURCE_OF_TRUTH_ROWS` in `apps/loopover-ui/src/lib/selfhost-docs-audit.ts`) is the + in-repo precedent for exactly this page→source mapping, already covering the 20 self-hosting pages; + this sweep's own page list should start from (and extend) that table rather than inventing a second one. +2. **Model**: a single AI pass per page — read the page's prose claims, read the mapped source file(s), + report ONLY claims it can affirmatively contradict against currently-read source, each with a + `file:line` citation. No `file:line` ⇒ no finding. This mirrors the issue's own hard requirement + ("no finding without a real file:line citation, no speculative 'this might be stale'") and the + existing AI-review call sites' shape: bounded, structured output, never free-form prose fed straight + to a human. +3. **Output**: one `DocsDriftReportRecord` (new table, mirroring `upstream_drift_reports`' shape — + `open`/status, a stable fingerprint per distinct claim, the page path, the cited `file:line`, the + claim text, and the contradiction) per confirmed finding, written the same way + `buildUpstreamDriftReport` (`ruleset.ts:335`) computes a report and `upsertUpstreamDriftReport` + (`src/db/repositories.ts:1665`) persists it today. + +## Scope-bounding (explicit requirement) + +- **N pages per run, rotating**, not all ~35 docs pages from day one. Cron enqueues a fixed-size slice + (e.g. 3-5 pages) each cycle, advancing a persisted cursor over the page list — the same + "bounded-batch, cursor-advances-every-run" shape `repo-doc-refresh-sweep` already uses for its own + per-repo eligibility fan-out (`src/index.ts:250-256`), just walking a static page list instead of a + dynamic repo list. +- **Cadence**: monthly, per the issue's own suggestion — checked on the existing hourly cron tick with an + `hour === X && dayOfMonth === Y` guard, mirroring `repo-doc-refresh-sweep`'s `hour === 9` daily guard + and the maintainer-recap job's day/hour cadence check (`src/index.ts:264-269`). A monthly full lap over + a 5-page/run, ~35-page corpus completes in ~7 cycles — well within a month at that cadence, so the + "bounded cost, predictable" requirement holds without needing a larger per-run batch. +- **One consolidated issue per finding-*run***, not per finding: exactly `fileUpstreamDriftIssues`'s own + shape (one row aggregating that run's findings → one fingerprinted issue, updated on the next run if + still open, never duplicated). + +## Non-goals (explicitly out of scope for this sweep) + +- Replacing or duplicating either mechanical sibling — this only covers claims neither can verify + mechanically. +- Auto-editing docs pages. The sweep only files a findings issue for a human to act on, same as the + upstream-drift pipeline never auto-edits `.loopover.yml`. +- A new AI-orchestration framework. The detection step is one bounded call per page through the existing + `SelfHostAi` abstraction, not a new agentic loop. + +## Open questions for maintainer sign-off before Phase 3 implementation begins + +1. Confirm the page→source mapping seed list (extend `SELFHOST_DOCS_PAGES`'s 20 pages to the full ~35-page + corpus, or scope this sweep to the self-hosting subset only for its first cut). +2. Confirm N (pages/run) and the monthly cadence, or adjust for cost/time predictability. +3. Confirm the new `docs_drift_reports` table + `LOOPOVER_AUTO_FILE_DOCS_DRIFT_ISSUES` flag naming, to keep + parity with the `upstream_drift_reports` / `LOOPOVER_AUTO_FILE_DRIFT_ISSUES` precedent this whole design + leans on.