fix(ci): Frontend Tests (unit) sat outside the fleet's only required check (#190)

[rubenvdlinde]

The issue's premise no longer holds — measured before changing anything

#190 says "0 of 20 fleet repos gate on Playwright, and 17 have no branch protection at all". Re-measured across 25 repos on 2026-08-08:

• All 25 have ACTIVE rulesets on main, beta and development. The "17 with no protection" figure is stale — the legacy /branches/*/protection API reads mostly empty because the gating moved to rulesets.
• Main Branch Protection and Beta Branch Protection each require exactly two contexts: branch-protection / check-branch and quality / Quality Report.
• Development Branch Protection requires zero checks.

development is unprotected by design and stays that way. Admin merges into development are this team's routine, authorised workflow; protecting it would block the thing the team does every day. Nothing in this PR touches it. Recording that here so it does not get "fixed" later.

E2E already gates main and beta — verified on real runs, not read off the YAML

Quality Report is not a pass-through. It carries if: ${{ always() && contains(needs.*.result, 'failure') }} → exit 1.

repo	run	Playwright	Quality Report
pipelinq	31181419092	failure	failure
scholiq	31164073861	failure	failure
nldesign	31252937998	failure	failure

The one counter-example — pipelinq 31181387613, Playwright cancelled, Quality Report success — is not a hole. That run's head SHA ce44766c was superseded by 47a1a590 twenty-nine seconds later, and the cancelled-job guard deliberately stays quiet for a superseded run because the newer run carries the verdict. The newer runs on 47a1a590 did report failure. (Checking whether the smoking gun also appears in a legitimately passing scenario is what stopped this becoming a wrong bug report.)

Why no explicit E2E context was added

Requiring quality / E2E Tests (Playwright) would be redundant with a gate measured to work, and it carries the failure mode this fleet has already been bitten by: four repos — openklant, opentalk, openzaak, shillinq — leave enable-playwright UNSET, so the job is SKIPPED there. A required check that does not report leaves PRs permanently pending, which is worse than no protection because it looks like caution.

Branch protection is outward-facing. Adding a duplicate context that can wedge four repos' merge queues is not an improvement over a gate that already works.

What was actually broken

Quality Report's needs: named fifteen jobs. The workflow has sixteen. frontend-tests — Frontend Tests (unit) — was not in it.

Both gating steps read needs.*.result. A job absent from needs: is absent from needs.*. So a real unit suite, with no continue-on-error, could fail — or be cancelled mid-run — and the only meaningful required check on main and beta across all 25 repos would stay green.

A failing frontend unit suite could not block a merge to main or beta anywhere in the fleet.

Latent, not active — and said so

Sampled across ten repos, Frontend Tests (unit) is green in every run (with some supersession cancellations). A search for a run with Frontend Tests (unit) = failure alongside Quality Report = success found none. Nothing has been waved through yet.

A gate that would not catch a failure is still a dead gate, and "it has not bitten yet" is not a verdict.

The fix and the guard

frontend-tests joins the list, and scripts/assert-quality-report-gates-every-leg.py asserts that every top-level job is either the report job or inside its needs:. An exemption needs an ALLOWLIST entry with a stated reason, so the list can only shrink and it says why.

Proven in both directions against the real document:

against origin/main   -> FAILS, naming 'frontend-tests'
against this commit   -> passes (all 16 legs gated)
--positive-control    -> drops a job from the list in memory and must notice; it names the job it dropped

It also refuses to pass on a degenerate parse (< 5 jobs) or a missing report job — the latter being its own outage, since renaming that job would make quality / Quality Report stop reporting and wedge every PR in the fleet.

It runs as its own job in quality-resolve-probe.yml behind the same single Shared-workflow guard required check, with its positive control first, and it lives in .github's own CI so it cannot itself go permanently pending.

What consumers see on the next run

The fleet tracks @main unpinned (#177), so this reaches all callers immediately.

• enable-frontend: false → job SKIPPED → not failure → nothing changes.
• Suite passes → nothing changes.
• Only a genuine frontend unit failure, or a frontend-tests job cancelled while its run carries on, newly blocks a merge to main or beta. That is the point.

No ruleset was modified in any repo. No waiver, no continue-on-error, no weakened assertion.

Refs #190