feat(quality): stop pinning the gates package per repo, and gate main on the failure mode that has no red by rubenvdlinde · Pull Request #177 · ConductionNL/.github

rubenvdlinde · 2026-08-06T05:22:07Z

What the PO asked for

can we stop pinning the packadge per repro causing al repos to alwyas use the newest packadge.

23 repositories pin hydra-gates-ref (21 on v1.3.0, openregister and softwarecatalog on v1.4.0). Every gate fix therefore needs 23 PRs to reach the fleet, and the bump PRs are a treadmill: 11 are open right now for v1.3.0 -> v1.4.0 alone.

Ref strategy: `main`, not a moving `v1`

The hydra-gates-ref default here was already main. This PR makes that the documented contract, removes the pins on the caller side (separate PRs, one per repo), and states the rollback levers.

A moving major tag was considered. It buys a release boundary — but only while someone remembers to re-point it, and a tag nobody re-points is indistinguishable from the stale pin this change exists to remove. It would reproduce the failure silently, which is the one property being engineered out. main cannot rot.

Rollback:

fleet-wide, immediate — revert the offending commit on this repo's main. One commit, no per-repo PRs, picked up on the next run. Strictly faster than the 23-PR bump a pin makes necessary.
one repo, immediate — set hydra-gates-ref: explicitly in that repo's caller. Still supported and still honoured; a temporary escape hatch, not a resting state.

Why main now needs a guard

Unpinning means a broken main reaches 23 repos at once. The failure mode that matters does not announce itself:

An unresolvable reusable workflow is not a red check. GitHub produces a run with no jobs, or no run at all. Nothing fails, every dashboard stays green, and every caller quietly stops being checked.

That is #161 — a ~22 KB run: step — reverted by #166, re-landed by #168. The signature is jobs == 0, not conclusion == failure.

file	role
`quality-selftest.yml`	in-repo caller of `./.github/workflows/quality.yml`. Exists to be resolved, not to pass. Its conclusion is not a verdict — do not make it a required check.
`quality-resolve-probe.yml`	counts the jobs that caller materialised, and asserts `> 0`. Never reads a conclusion.
`scripts/count-workflow-jobs.sh`	the counter. Polls, so a scheduling race is not mistaken for the outage. Always exits 0 — the caller judges, so the instrument can be controlled against an expected-zero case.
`scripts/assert-run-steps-resolvable.py`	static lint: no `run:` block over 16 KB. Catches the #161 class before merge and names the offending step, which a job count cannot.

Two workflows, deliberately. With the uses: call inside the probe, an unresolvable quality.yml would delete the probe too — the required check would never report and the PR would sit pending forever, the hardest of the three dead-gate shapes to notice. Split in two, the probe always runs and the outage surfaces as an explicit red.

Proof the guards can fail

Both are positive-controlled on every run, not once at authoring time:

the probe first counts a workflow that cannot exist and requires the answer to be 0. A counter that had started returning a constant, or that had lost its token scope, would otherwise pass the real check forever.
the lint is re-run at a 1-byte limit it cannot possibly meet.

And the lint reproduces the real historical outage. Run against quality.yml as it stood at d68fb72 (the commit that took the fleet down):

::error file=q161.yml,line=2338::`run:` block is 26959 bytes, over the 16384-byte limit.
rc=1

Against current main: OK — every run: block is under 16384 bytes, rc=0. Threshold sits above the largest step the fleet ships (10533 bytes) and below the size known to have broken it.

`hydra-gates-package.yml` loses its `paths:` filter

Defensible while callers pinned — a repo on v1.3.0 was unaffected by anything landing here. Not defensible once the fleet tracks @main: the runner resolves at run time, so a PR that touches nothing under hydra-gates/ can still be the commit a caller picks up. A required check with a paths: filter is also its own trap — every excluded PR sits pending. The alternative was a hand-maintained filter, which decays exactly like the hand-maintained test list this same workflow already learned not to keep.

Refs #159, #161, #166, #168, #173

@main

… on the failure mode that has no red The fleet pinned `hydra-gates-ref` in 23 repositories. A pin is a silent expiry date, and it has now cost us twice: #159 22 repos sat on v1.0.1, predating a batch of gate fixes. 16 gates were dead fleet-wide and every one reported PASS — a gate whose helper never runs emits a tick identical to a real one. #173 the coverage flag was flipped to default true here, at @main, and reached v1.0.1 runners predating the accounting that makes it survivable. decidesk and doriath went red (exit 98) on gates 4, 24 and 33 — the not-applicable gates that must never fail a run. Both are one defect: this workflow is consumed at @main while the package was consumed at a pin, so the two sides moved independently and a change on one reached a runner from the other's past. The fix is no pin. The default was already `main`; this makes that the documented contract and states the rollback levers, because "always latest" is only safe with a way back: revert on main for the whole fleet, or set the input explicitly for one repo. A moving `v1` tag was considered and rejected — a tag nobody re-points is indistinguishable from the stale pin being removed. Unpinning means a broken main now reaches 23 repos at once, so main is gated on the one failure mode that does not announce itself. An unresolvable reusable workflow is not a red check: GitHub produces a run with no jobs, or no run, and every dashboard stays green while every caller quietly stops being checked. That is what a ~22 KB `run:` step did in #161, reverted by #166 and re-landed by #168. quality-selftest.yml an in-repo caller of ./.github/workflows/ quality.yml. It exists to be RESOLVED, not to pass; its conclusion is not a verdict. quality-resolve-probe.yml counts the JOBS that caller materialised. `jobs == 0` is the outage signature, so the count is the measurement and a conclusion is deliberately not read. Split across two files on purpose: with the call in the probe itself, an unresolvable quality.yml would delete the probe too and the required check would sit PENDING FOREVER rather than go red. Both guards are positive-controlled on every run. The probe first counts a workflow that cannot exist and requires the answer to be 0 — a counter returning a constant would otherwise pass forever. The lint is re-run at a 1-byte limit it cannot meet. The lint also reproduces the real outage: against quality.yml at d68fb72 it names the 26959-byte block at line 2338. hydra-gates-package.yml loses its `paths:` filter. It was defensible while callers pinned; with the fleet on @main the blast radius of any merge here is the whole fleet, and a required check with a paths filter leaves every excluded PR pending forever. Refs #159, #161, #166, #168, #173

…econd silent way to zero jobs The selftest shipped with `contents: read`. Run 31073908079 came back startup_failure with ZERO jobs, because quality.yml's journeydoc-capture job declares `contents: write` + `actions: write` and permissions are validated when the run is CREATED — `enable-journeydoc-capture: false` never gets a chance to matter, since the job's `if:` is not reached. This is a second route into the #161 outage class and it applies to every caller in the fleet: a permission grant narrower than the shared workflow requests produces no jobs and no red, indistinguishable from a healthy run on any dashboard. The probe caught it on its first execution, which is the evidence that it works. Run 31073907988 recorded, in order: control count = 0 OK — the counter can return 0, and reports it. quality-selftest.yml materialised 0 jobs at 7c3d160 ##[error]quality.yml produced ZERO jobs. This is the #161 outage signature So the guard has now been shown to fail on a REAL zero-jobs condition, not only against its synthetic control.

…g one state too far (#180) * fix(gate-28): an EMPTY DIFF SCOPE is not a structural gap — #172 swung one state too far #172 was right that gate-28 must not report PASS having opened zero files. But there are TWO ways to read zero files and it gave them the same word: (a) lib/**/*.php files ARE in this diff and none carries a licence tag — a genuine gap in the repository, correctly `structural`; (b) no lib/**/*.php file is in this diff AT ALL — diff-scoping working exactly as ADR-020 designed it, and no statement about the repo. (b) is the ordinary case. `_in_scope` filters to the PR's diff, so every workflow-only and frontend-only PR reads zero lib PHP files. Calling that `structural` counts it against coverage, and with hydra-gates-require-full-coverage on by default the run fails with exit 98 for a licence problem the repository does not have. MEASURED on the fleet's own unpinning PRs, each a single-file workflow diff: hrmq#74 CLEAN baseline -> RED on nothing but gate-28 app-versions#129 gate 28 its only gate that did not run opencatalogi#813 " nextcloud-app-template#132 " launchpad#60 " hrmq's 168 lib PHP files all carry their tags. The gate had simply not been handed any of them to read, and then reported that as the repository's fault. This is the "a gate that is legitimately not applicable must NOT fail the run" rule broken — the same shape as #173, one gate lower down. It surfaced now because the fleet just unpinned (#177): at v1.3.0 gate-28 reported PASS on an empty read, so the false GREEN hid what is now a false RED. Neither was a verdict. The `structural` branch is preserved for case (a) and now names the count of files that WERE in scope, so the two states no longer read alike. TESTS — scripts/lib/test_gate_license_triangle_scope.sh, four control pairs, discovered automatically by tests/run-helper-suites.sh: wf-only diff, tagged repo -> NOT APPLICABLE (b, the false red) untagged PHP in the diff -> structural (a, must stay red) tagged PHP in the diff -> PASS (the gate compared something) no lib/ at all -> NOT APPLICABLE (#172's original case) Verified in BOTH directions rather than assumed: against the fixed runner 5/5 green; against the runner at 954a78a the first two FAIL and the last three still pass, so the suite discriminates and the fix is targeted rather than a mute for #172. The did-not-run assertion took two attempts to measure anything — v1 matched only a summary line this suite never triggers, v2 matched a roster the runner also prints for NOT-APPLICABLE gates. Both are recorded in the file, because a dead assertion inside a suite meant to tell two states apart is the exact defect the suite exists to catch. * fix(test): explicit if/else in _expect — `A && _ok || _bad` can count one assertion twice ShellCheck SC2015 on the new suite, and it is not a style note here. In `grep -q ... && _ok "$d" || _bad "$d"`, a non-zero return from _ok makes _bad run as well, so the SAME assertion is counted once as a pass and once as a failure. In a harness whose entire job is to count assertions correctly, that is instrument corruption — the class this suite exists to catch, one level up. Also adds a `*)` arm: an unknown expectation string now fails loudly instead of matching no case and silently asserting nothing. Re-verified in both directions after the change: fixed runner 5 passed / 0 failed, runner at 954a78a 3 passed / 2 failed, entry-point invariants 39/0. --------- Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

…it was cut (#331) This repo has been bumped twice already (v1.0.1 -> v1.3.0 -> v1.5.0), each time to clear a breakage the pin itself caused, and a third was queued. v1.5.0 ships gate-28 reporting a `structural` coverage gap whenever the diff scope is empty — i.e. on every PR that does not touch lib/**/*.php, which is most of them. With hydra-gates-require-full-coverage defaulting TRUE that is exit 98. It is fixed in ConductionNL/.github#182, on main, and a pinned repo cannot see that fix because the tag will not move. Tracking main is the fleet contract as of ConductionNL/.github#177, which also added the guard that makes it safe: an unresolvable reusable workflow produces no red check at all, just zero jobs, so quality-resolve-probe.yml counts the jobs a caller materialises and asserts > 0. Supersedes #329, which set v1.4.0 — a DOWNGRADE from what development now carries, and to a ref that fails #177's capability probe outright (v1.4.0 ships none of check_spec_anchors.py, check_form_labels.py or check_license_triangle.py). Rollback for this repo alone stays available: set hydra-gates-ref explicitly. Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

@main

…check (#190) (#229) MEASURED FIRST — the issue's premise no longer holds #190 says "0 of 20 fleet repos gate on Playwright, and 17 have no branch protection at all". Re-measured across 25 repos on 2026-08-08, the second half is simply false and the first half is misleading: * ALL 25 repos have ACTIVE rulesets on main, beta AND development. * `Main Branch Protection` and `Beta Branch Protection` each require exactly two contexts: `branch-protection / check-branch` and `quality / Quality Report`. * `Development Branch Protection` requires ZERO checks. That is correct and deliberate — development is unprotected BY DESIGN, because admin merges into it are this team's routine workflow. Nothing here changes it. And E2E already gates main and beta, transitively. `Quality Report` is not a pass-through: it exits 1 on `contains(needs.*.result, 'failure')`. Verified on real runs rather than read off the YAML — pipelinq 31181419092 Playwright failure -> Quality Report failure scholiq 31164073861 Playwright failure -> Quality Report failure nldesign 31252937998 Playwright failure -> Quality Report failure The one counter-example, pipelinq 31181387613 (Playwright CANCELLED, Quality Report SUCCESS), is not a hole: that run's head SHA ce44766c was superseded by 47a1a590 twenty-nine seconds later, and the cancelled-job guard deliberately stays quiet for a superseded run because the newer run carries the verdict. The newer runs on 47a1a590 did report failure. So no new required context was added. Requiring `quality / E2E Tests (Playwright)` explicitly would be redundant with a gate measured to work, and it carries the risk the fleet has already been bitten by three times: four repos (openklant, opentalk, openzaak, shillinq) leave `enable-playwright` UNSET, so the job is SKIPPED there, and a required check that does not report leaves PRs permanently PENDING — worse than no protection because it looks like caution. WHAT WAS ACTUALLY BROKEN `Quality Report`'s `needs:` list named fifteen jobs. The workflow has sixteen. `frontend-tests` ("Frontend Tests (unit)") was not in it. Both of that job's gating steps read `needs.*.result`. A job absent from `needs:` is absent from `needs.*`, so `Frontend Tests (unit)` — a real unit suite with no `continue-on-error` — could fail, or be cancelled mid-run, and the only meaningful required check on main and beta across all 25 repos would stay green. A failing frontend unit suite could not block a merge to main or beta anywhere in the fleet. LATENT, NOT ACTIVE, AND SAID SO. Sampled across ten repos the job is currently green in every run (with some supersession cancellations), and a search for a run with `Frontend Tests (unit) = failure` alongside `Quality Report = success` found none. Nothing has been waved through yet. A gate that would not catch a failure is still a dead gate, and "it has not bitten yet" is not a verdict. THE FIX AND THE GUARD `frontend-tests` joins the list, and `scripts/assert-quality-report-gates-every-leg.py` now asserts that EVERY top-level job is either the report job or inside its `needs:`. An exemption requires an ALLOWLIST entry with a stated reason, so the list can only shrink and it says why. Proven in both directions, against the real document: * run against `origin/main` -> FAILS, naming `frontend-tests` * run against this commit -> passes * `--positive-control` drops a job from the list in memory and requires the check to notice; it names the job it dropped It runs as its own job in quality-resolve-probe.yml, behind the same single `Shared-workflow guard` required check, with its positive control first — and it lives in .github's own CI, so it cannot itself go permanently pending. BLAST RADIUS FOR CONSUMERS The fleet tracks @main unpinned (#177), so this reaches all callers on their next run. Repos where `enable-frontend` is false see a SKIPPED job, which is not 'failure' and changes nothing. Repos where the suite passes see nothing. Only a genuine frontend unit failure — or a frontend-tests job cancelled while its run carries on — newly blocks a merge to main or beta. That is the point. Refs #190 Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

Conduction Release Bot added 2 commits August 6, 2026 07:21

rubenvdlinde merged commit 954a78a into main Aug 6, 2026
29 checks passed

rubenvdlinde mentioned this pull request Aug 6, 2026

ci(quality): unpin hydra-gates — a pin cannot carry a fix made after it was cut ConductionNL/zaakafhandelapp#331

Merged

This was referenced Aug 8, 2026

fix(ci): a failing seed exits 0 (#192); a tag inside describe.skip counts as live (#210) #212

Merged

fix(ci): Frontend Tests (unit) sat outside the fleet's only required check (#190) #229

Merged

Navigation Menu

Uh oh!

feat(quality): stop pinning the gates package per repo, and gate main on the failure mode that has no red - #177

feat(quality): stop pinning the gates package per repo, and gate main on the failure mode that has no red#177
rubenvdlinde merged 2 commits into
mainfrom
chore/track-latest-gates

rubenvdlinde commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rubenvdlinde commented Aug 6, 2026

What the PO asked for

Ref strategy: main, not a moving v1

Why main now needs a guard

Proof the guards can fail

hydra-gates-package.yml loses its paths: filter

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Ref strategy: `main`, not a moving `v1`

`hydra-gates-package.yml` loses its `paths:` filter