revert(quality): restore fleet CI — #161 made quality.yml unresolvable, every caller startup-failed with ZERO jobs - #166
Merged
Conversation
rubenvdlinde
added a commit
that referenced
this pull request
Aug 5, 2026
…22 KB `run:` step is what made the workflow unresolvable (#168) #161 scoped axe to the app's own DOM and was reverted in #166 because every caller in the fleet started up with ZERO jobs and a run `name` equal to the raw file path — GitHub never RESOLVED quality.yml. The revert was right. This re-lands the same measured behaviour in a shape GitHub resolves, and names the cause rather than guessing at it. CAUSE, bisected with a controlled probe (a `uses: ./…` caller inside this repo, so resolution can be tested on a branch without pointing an app repo at unmerged code). Read the JOB COUNT, never the conclusion — `failure` with jobs>0 means it resolved and something inside failed, which is a PASS for the probe: baseline (current main) -> 25 jobs RESOLVES #161 re-applied in full -> 0 jobs outage reproduced 45 workflow_call inputs, step untouched -> 25 jobs RESOLVES whole FILE padded to 226 KB, step untouched -> 25 jobs RESOLVES ONE `run:` step padded to 21 KB with INERT JS COMMENTS, nothing else changed -> 0 jobs BREAKS So it is neither the input count (43 -> 45) nor the file size, the two candidates named in the revert. It is the size of a SINGLE `run:` step, and the padding control proves it is size rather than anything in the content. Bracketed: 18,965 bytes resolves, 21,109 bytes does not. #161's axe step was 22,439. FIX: the axe runner is now hydra-gates/scripts/axe-run.cjs — a real file, byte identical to the one proven under #161 (md5 40ba28f6…) — checked out into the Playwright job the same way the hydra-gates job already checks this repo out. The axe step drops 22,439 -> 2,162 bytes, the largest `run:` step in the whole file is now 8,533 bytes, and quality.yml is 201,108 bytes: SMALLER than the 207,576 it was before #161, because the program moved out of the YAML. A 500-line Node program embedded in a YAML string was the real defect. Keeping it in a file takes the entire class of failure off the table instead of leaving the next comment one edit away from another fleet outage. The behaviour is unchanged from #161 and re-verified against the shipped file: openregister 2 serious/critical -> 0 scoped, every violation removed provably core's; a violation injected INSIDE the container still reported at critical; the same violation OUTSIDE not reported; a scope matching nothing aborts without writing a report. Guard 4 asserts both halves on the live page every run. Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
rubenvdlinde
added a commit
that referenced
this pull request
Aug 5, 2026
…ed runners — decidesk and doriath went red on gates 4, 24 and 33 (#173) `hydra-gates-require-full-coverage` was flipped to default TRUE (#164) once the runner could tell a not-applicable gate from a real gap. But this workflow is consumed at @main while the gate package is consumed at the caller's own `hydra-gates-ref`, so the new default was handed to runners that predate the accounting entirely — and those count EVERY gate they did not run as a gap. Measured, not inferred, both after the flip and both failing on nothing else: decidesk job 92432089644 ref v1.0.1 ALL 61 GATES GREEN COVERAGE: 58 of 61 / DID NOT RUN: 4 24 33 -> exit 98 doriath job 92363547732 ref v1.0.1 same three gates, same verdict Gates 4, 24 and 33 are the exact trio the product owner named as legitimately not applicable. So the flip was producing the outcome it was made to prevent, in every repo whose pin had not moved — and pins do not move on in-flight branches. The flag is now handed only to a package that can honour it. The probe reads the package's own accounting (`_NA_GATES` in the runner, the `NOT APPLICABLE` verdict in the wrapper) rather than comparing versions, because a pin may be a tag, a branch, a SHA or a fork. Verified against every published tag: v1.0.0/v1.0.1/v1.1.0/v1.2.0 -> withheld, v1.3.0/v1.4.0 -> applied. A checkout missing either file is a broken checkout, not an old one, and fails. Withholding is stated as a `::warning::` naming the pin and the fix. A control that switches itself off quietly is a control that has stopped existing. `--axe-enabled` is gated on the same probe. The old wrapper's parser ends in `*) APP_DIR="$1"`, so an unrecognised flag is swallowed as a POSITIONAL and clobbers the app dir set moments earlier: a v1.0.1 pin handed it aborts with `FATAL: --app-dir '--axe-enabled' is not a directory. No gate ran.` — exit 99, every gate lost, for a flag the caller never typed. Same ref skew, second instance: axe-run.cjs landed in v1.4.0 and every caller is pinned v1.3.0, so `enable-axe: true` would die on `Cannot find module` and then present as "the axe report never arrived" — pointing at Playwright, the one place the fault is not. Asserted at the checkout with a named error instead. The two probe literals are a CONTRACT: renaming either would silently disable coverage enforcement fleet-wide, with a warning blaming the caller's pin. The package's own suite now asserts both, plus a reverse control proving the probe does not match the pre-v1.3.0 shape of either file. Positive-controlled: with both literals renamed in a copy, those assertions go red. Four-way measurement on one fixture, one variable at a time: v1.0.1 + --require-full-coverage exit 98 <- the observed failure v1.0.1, flag withheld (this change) exit 0 v1.4.0 + flag, only (a) gaps exit 0 33 of 33 applicable gates ran v1.4.0 + flag + a structural gap exit 98 [gate-24] SKIPPED (structural) Also removes a stale comment left by the #161/#166/#168 cycle that instructed future readers to embed the axe runner inline — the thing that made the whole workflow unresolvable fleet-wide. Suite 31 -> 34 tests, 0 failing. actionlint clean (positive-controlled: a typo'd step id is reported). Largest `run:` step 8,533 B, far below the ~19 KB that makes a workflow unresolvable. Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
rubenvdlinde
added a commit
that referenced
this pull request
Aug 6, 2026
… on the failure mode that has no red (#177) * feat(quality): stop pinning the gates package per repo, and gate main on the failure mode that has no red The fleet pinned `hydra-gates-ref` in 23 repositories. A pin is a silent expiry date, and it has now cost us twice: #159 22 repos sat on v1.0.1, predating a batch of gate fixes. 16 gates were dead fleet-wide and every one reported PASS — a gate whose helper never runs emits a tick identical to a real one. #173 the coverage flag was flipped to default true here, at @main, and reached v1.0.1 runners predating the accounting that makes it survivable. decidesk and doriath went red (exit 98) on gates 4, 24 and 33 — the not-applicable gates that must never fail a run. Both are one defect: this workflow is consumed at @main while the package was consumed at a pin, so the two sides moved independently and a change on one reached a runner from the other's past. The fix is no pin. The default was already `main`; this makes that the documented contract and states the rollback levers, because "always latest" is only safe with a way back: revert on main for the whole fleet, or set the input explicitly for one repo. A moving `v1` tag was considered and rejected — a tag nobody re-points is indistinguishable from the stale pin being removed. Unpinning means a broken main now reaches 23 repos at once, so main is gated on the one failure mode that does not announce itself. An unresolvable reusable workflow is not a red check: GitHub produces a run with no jobs, or no run, and every dashboard stays green while every caller quietly stops being checked. That is what a ~22 KB `run:` step did in #161, reverted by #166 and re-landed by #168. quality-selftest.yml an in-repo caller of ./.github/workflows/ quality.yml. It exists to be RESOLVED, not to pass; its conclusion is not a verdict. quality-resolve-probe.yml counts the JOBS that caller materialised. `jobs == 0` is the outage signature, so the count is the measurement and a conclusion is deliberately not read. Split across two files on purpose: with the call in the probe itself, an unresolvable quality.yml would delete the probe too and the required check would sit PENDING FOREVER rather than go red. Both guards are positive-controlled on every run. The probe first counts a workflow that cannot exist and requires the answer to be 0 — a counter returning a constant would otherwise pass forever. The lint is re-run at a 1-byte limit it cannot meet. The lint also reproduces the real outage: against quality.yml at d68fb72 it names the 26959-byte block at line 2338. hydra-gates-package.yml loses its `paths:` filter. It was defensible while callers pinned; with the fleet on @main the blast radius of any merge here is the whole fleet, and a required check with a paths filter leaves every excluded PR pending forever. Refs #159, #161, #166, #168, #173 * fix(selftest): widen the caller's permissions — a narrow grant is a second silent way to zero jobs The selftest shipped with `contents: read`. Run 31073908079 came back startup_failure with ZERO jobs, because quality.yml's journeydoc-capture job declares `contents: write` + `actions: write` and permissions are validated when the run is CREATED — `enable-journeydoc-capture: false` never gets a chance to matter, since the job's `if:` is not reached. This is a second route into the #161 outage class and it applies to every caller in the fleet: a permission grant narrower than the shared workflow requests produces no jobs and no red, indistinguishable from a healthy run on any dashboard. The probe caught it on its first execution, which is the evidence that it works. Run 31073907988 recorded, in order: control count = 0 OK — the counter can return 0, and reports it. quality-selftest.yml materialised 0 jobs at 7c3d160 ##[error]quality.yml produced ZERO jobs. This is the #161 outage signature So the guard has now been shown to fail on a REAL zero-jobs condition, not only against its synthetic control. --------- Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The outage
Every
code-qualityrun created across the fleet after 2026-08-05T13:29:41Z — the timestamp ofd68fb727(#161) — starts up with zero jobs and reportsfailure. Reproduced on openregister, opencatalogi, openconnector and docudesk, on branches with no changes under.github/.The two discriminators
This does not look like a broken pipeline from the rollup. It looks like an ordinary red, which is why it is dangerous:
nameis the raw path —.github/workflows/code-quality.yml— instead of the workflow'sname:(Code Quality). GitHub never resolved the workflow, so it never learned its name.conclusionisfailure. Anyone reading only the conclusion sees a normal failure and starts debugging their own diff.Also:
gh run rerunrefuses it ("workflow file may be broken"), andcreated_at == updated_at.Same repo, same callers, minutes apart:
nameCode Quality.github/workflows/code-quality.ymlWhy a revert rather than a forward fix
The cause is above YAML syntax — the file parses cleanly, including under a strict loader with duplicate-key detection — so the fix needs a controlled experiment rather than a guess, and the fleet cannot stay dark while that happens. #161's measured work is good and should return; this only removes it from
mainuntil it can be re-landed in a form GitHub resolves.The change added two
workflow_callinputs (43 → 45) and grew the file 207,576 → 224,420 bytes. Both are candidates and neither is yet proven.Verification before merge
A caller re-run must produce a run whose
nameisCode Qualitywith a non-zero job count. Aconclusionoffailureis not sufficient evidence either way here — the job count is the discriminator.Refs #155.