US-32.1 AC-32.1.2: guard the revoke sweep's plan in the scale job

[mkreyman]

Replaces #934. #934's third review round still found material defects, so under the review ceiling this is a rewrite rather than a fourth round. It follows up #932, which removed the default suite's plan-choice assertion for AC-32.1.2.

What it asserts: RevokeExpiredDispatchesPlanScaleTest (:scale, run by a new step in CI's scale-test job):

• Commits a dispatches table in the shape a running one settles into: a large revoked history, a small live set, and a small expired backlog.
• Rebuilds the partial index and runs VACUUM ANALYZE.
• Captures the query RevokeExpiredDispatchesWorker.perform/1 actually issues, through repo telemetry.
• Requires the default planner to use dispatches_expires_at_active_index, through the existing PlanAssertions.assert_index_used/2.

The query reads one table, so a plan that uses the index cannot also Seq Scan it. With 50 live rows the planner uses an Index Scan. With a larger live set it uses a Bitmap Index Scan. Both seek the index. Measured on a clean table, the index is used from 50 live rows up to half the table.

What #934 got wrong, and this doesn't:

• Leaking cleanup: US-32.1 AC-32.1.2: guard the revoke sweep's natural plan in the scale job #934's cleanup never finished. The unindexed self-reference parent_dispatch_id makes a plain delete of 20k dispatches scan the table once per row, so every local run leaked its seed. About 200k rows piled up, and that skewed the plan measurements US-32.1 AC-32.1.2: guard the revoke sweep's natural plan in the scale job #934 recorded.
  • Here the dispatches are deleted with FK triggers off, inside their own transaction.
  • The tenant is deleted afterwards, with its cascades on.
  • The slug is a UUID, so a leftover tenant can't collide with the next run's.
• Bloated index: in a database the test had already run in, the partial index held 70 entries in 131 pages. The index scan and the Seq Scan then cost nearly the same, and the plan flipped from run to run. The seed now runs REINDEX on the index and VACUUM ANALYZE, so every run plans against the state CI's fresh database has.
• Smaller change: no new PlanAssertions helper, no CI-only skip, and no database-name guard. It is tagged :scale like every other scale test.

Evidence:

• Stable: 6 of 6 local runs passed, with a pause in the middle. No rows are left afterwards.
• Mutations (mutate.sh, all exit 0):
  • worker predicate made non-implying (coalesce)
  • range on expires_at replaced by a date cast
  • asserted index name swapped
• Gate: 12002 tests, 0 failures.

Review round 1 found 10 issues. All 10 are fixed:

• Index at the cause: the cleanup worked around an unindexed foreign key. Migration 20260928140000 now adds concurrent partial indexes on both unindexed foreign keys into dispatches:

  • dispatches.parent_dispatch_id
  • story_acceptance_criteria.verified_by_dispatch_id

  Every bulk delete of dispatches used to scan the referencing table once per row. The test's cleanup is now a plain delete with triggers on. ForeignKeyIndexesTest fails the default suite whenever a foreign key into dispatches has no index.

• Sweep rolled back: the sweep is cross-tenant, so it now runs in a transaction the test rolls back. Inside that transaction the test checks that exactly the backlog was revoked.

• Composite ruled out: PlanAssertions.refute_index_used/2 rules out the tenant-leading composite index, so a bitmap scan that reads it whole fails.

• Smaller fixes:

  • the captured query is matched by SELECT ... FROM "dispatches", not by one spelling of its predicate;
  • the index rebuild uses REINDEX CONCURRENTLY;
  • the tenant comes from fixture(:tenant);
  • the moduledoc names the async exception;
  • both independent scale-job steps run unless the job was cancelled;
  • AC-32.1.2 and the README agree on the accepted scans.

Mutations (mutate.sh, each exit 0):

• index dropped under ForeignKeyIndexesTest
• worker predicate made non-implying
• asserted index swapped
• composite check pointed at the partial index
• the sweep's write removed

Gate: 12003 tests, 0 failures. The first attempt hit one 57014 in KnowledgeAutoExtractTest: an insert into a table that references tenants was cancelled. That is the same unconfirmed lock-wait flake recorded in #932. The retry passed.

Review round 2 found 9 issues. All 9 are fixed:

• FK check: exact leading columns.
• Plan check: the named index must exist and be the only index in the plan.
• Migration: reconciles with stale?/2, so a re-run never drops a valid index.
• Seed timing: live rows expire hours out.
• Leftovers: invalid rebuild leftovers are dropped first.
• CI: the gates no longer run after a failed setup.
• Fixture and status: the seed is a fixture, and US-32.1's criteria are marked complete.

Review round 3, the last, found 10 issues. All 10 are fixed. None was in the migration or in what the guard catches; all were about how precise the helper checks are:

• FK check: counts only btree key columns (not INCLUDE columns), and accepts a partial index only when its NOT NULL predicate is on the foreign key's own column.
• Statistics freshness: judged on the column the planner uses, by pg_stats.null_frac against the actual NULL fraction.
• Index validity: the index must be valid and on the named table.
• Migration shape: the regex is anchored.
• Backlog timing: backlog rows expire minutes in the past, clear of clock skew.
• Leftovers: the leftover filter is exact.
• CI: the gates also require the grant step.

Mutations (each caught):

• FK index dropped
• FK index replaced with one that has another column's NOT NULL predicate
• FK index replaced with one that has the key only as an INCLUDE column
• FK index replaced with a hash index
• worker predicate made non-implying
• a second index in the plan
• the wrong table
• a differently shaped seed without ANALYZE: stale null_frac 0.0035 against an actual 0.20
• the sweep's write removed

Gate: 12003 tests, 0 failures.