US-26.4.6: story verification reads CI from the intake source, per-tenant credential (fixes #913)

[mkreyman]

Fixes #913. Implements story US-26.4.6 (docs/user_stories/epic_26_chain_of_custody_v2/us_26.4.6.json, #916). Supersedes #914, #915 and #931.

Approved by Mark on 2026-09-28 after round 3, together with one extra review round. That round ran, and its findings are fixed in the last commit.

Problem:

• Story verification read CI through GET /commits/:sha/check-runs. That API needs checks: read, which a fine-grained token can't have.
• It also read the tenant-controlled projects.repo_url with the operator token, which is a cross-tenant oracle.
• On master, an uncast schemaless query crashes every started run before it reaches the CI read. That crash is the only reason neither problem is live today.

What verification does now:

• Repository and branch: the repository comes only from the project's intake source. The branch is the story's own: the thread branch, or the placed pr branch.

• Credential: the operator's GITHUB_TOKEN is used only for (tenant, repository) pairs listed in the new VERIFICATION_OPERATOR_TOKEN_TENANTS, which is empty by default. The checks run in this order:

  • a tenant with no entry records credential_unavailable before anything else is read;
  • a pr source that names no required checks records no_required_checks;
  • the pair is checked before any GitHub read.

  Per-tenant GitHub credentials are Per-tenant GitHub credentials for forge reads (merge gate + verification) #936.

• Change check, once per run:

  • a branch that changes its own CI definition is refused ci_definition_changed;
  • an empty change is refused empty_change: an empty three-dot diff, or a head tree equal to the base tree.

  Both are the merge gate's own rules, shared through CiDefinition and the new EmptyChange. The check's pass is stamped on the run (change_checked_at), so a merge during the CI wait can't later refuse a checked commit.

• Verdict: GitHubPullRequestSource.check_evidence/3, judged by CiEvidence.judge/2 over the source's required_checks.

• Bounded waits:

  • Pending CI waits until the run's age limit, then records ci_wait_exhausted.
  • Every poll that ends in a transient forge fault counts toward the merge gate's consecutive bound, then records forge_unavailable. Only a CI answer resets the count.
  • Snoozes back off 60/120/240/480/900 s.
  • Database contention on any read or write snoozes as database_busy. It never touches the fault count and leaves the run unchanged.

• Abbreviated SHA: resolved once, then stored on the run. GitHub's 422 is recorded as unresolved_sha; its 404 as repository_unreadable.

• Recorded results: ac_results hold fixed codes only, with no GitHub-echoed URLs and no exception text.

• No local fallback: TestRunner and :enable_local_test_runner are deleted. The runner had no environment it could safely run in: it would execute tenant code with loopctl's environment and filesystem. AC-26.4.6.6 says what brings it back.

• Crash fixed: the uncast story query that crashed every started run is gone.

Shared with the merge gate: CiDefinition, EmptyChange, and in CiEvidence the rule that only a jobless run that concluded as a failure counts against a check.

Migrations: 20260928120000 adds resolved_commit_sha and ci_forge_faults, and 20260928150000 adds change_checked_at, both to verification_runs. Both are additive and nullable.

Review history: four rounds, the fourth approved by Mark.

• Round 1:
  • scrubbed the fallback's environment (the fallback was later removed);
  • moved the allowlist to pairs;
  • corrected the resolve codes and the jobless-run rule.
• Round 2:
  • removed the local fallback;
  • fixed the fault streak;
  • refused on-base commits.
• Round 3: tried binding verification to the commit the merge gate allowed. That rested on the wrong order. Verification runs before the merge, because the gate allows nothing until the story is verified.
• Round 4:
  • removed that binding;
  • moved to the shared empty-change rule;
  • made every worker write a bounded wait.

Evidence:

• Mutations (mutate.sh), each exit 0, across the rounds. The final set:
  • change check: stamp, once-skip, shared tree rule and empty diff through all three callers, tree wiring, unreadable arm, gate wiring;
  • credentials: pair order, allowlist, pair wiring;
  • verdict and waits: CI definition, no-verdict arm, fault streak, resolve once, MCP doc order;
  • bounded writes: one per wrapped write, plus busy class, lock bound and rollback.
• Gate: 12089 tests, 0 failures. MCP: 812 pass.