Skip to content

revert(quality): restore fleet CI — #161 made quality.yml unresolvable, every caller startup-failed with ZERO jobs - #166

Merged
rubenvdlinde merged 1 commit into
mainfrom
fix/restore-ci-revert-axe-scope
Aug 5, 2026
Merged

revert(quality): restore fleet CI — #161 made quality.yml unresolvable, every caller startup-failed with ZERO jobs#166
rubenvdlinde merged 1 commit into
mainfrom
fix/restore-ci-revert-axe-scope

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

The outage

Every code-quality run created across the fleet after 2026-08-05T13:29:41Z — the timestamp of d68fb727 (#161) — starts up with zero jobs and reports failure. Reproduced on openregister, opencatalogi, openconnector and docudesk, on branches with no changes under .github/.

The two discriminators

This does not look like a broken pipeline from the rollup. It looks like an ordinary red, which is why it is dangerous:

  1. The run's name is the raw path.github/workflows/code-quality.yml — instead of the workflow's name: (Code Quality). GitHub never resolved the workflow, so it never learned its name.
  2. Jobs = 0, while conclusion is failure. Anyone reading only the conclusion sees a normal failure and starts debugging their own diff.

Also: gh run rerun refuses it ("workflow file may be broken"), and created_at == updated_at.

Same repo, same callers, minutes apart:

created run name jobs
13:22:46Z Code Quality 28
13:33:34Z .github/workflows/code-quality.yml 0

Why a revert rather than a forward fix

The cause is above YAML syntax — the file parses cleanly, including under a strict loader with duplicate-key detection — so the fix needs a controlled experiment rather than a guess, and the fleet cannot stay dark while that happens. #161's measured work is good and should return; this only removes it from main until it can be re-landed in a form GitHub resolves.

The change added two workflow_call inputs (43 → 45) and grew the file 207,576 → 224,420 bytes. Both are candidates and neither is yet proven.

Verification before merge

A caller re-run must produce a run whose name is Code Quality with a non-zero job count. A conclusion of failure is not sufficient evidence either way here — the job count is the discriminator.

Refs #155.

@rubenvdlinde
rubenvdlinde merged commit 25e28fa into main Aug 5, 2026
3 checks passed
rubenvdlinde added a commit that referenced this pull request Aug 5, 2026
…22 KB `run:` step is what made the workflow unresolvable (#168)

#161 scoped axe to the app's own DOM and was reverted in #166 because every
caller in the fleet started up with ZERO jobs and a run `name` equal to the raw
file path — GitHub never RESOLVED quality.yml. The revert was right. This
re-lands the same measured behaviour in a shape GitHub resolves, and names the
cause rather than guessing at it.

CAUSE, bisected with a controlled probe (a `uses: ./…` caller inside this repo,
so resolution can be tested on a branch without pointing an app repo at unmerged
code). Read the JOB COUNT, never the conclusion — `failure` with jobs>0 means it
resolved and something inside failed, which is a PASS for the probe:

  baseline (current main)                        -> 25 jobs   RESOLVES
  #161 re-applied in full                        ->  0 jobs   outage reproduced
  45 workflow_call inputs, step untouched        -> 25 jobs   RESOLVES
  whole FILE padded to 226 KB, step untouched    -> 25 jobs   RESOLVES
  ONE `run:` step padded to 21 KB with INERT
    JS COMMENTS, nothing else changed            ->  0 jobs   BREAKS

So it is neither the input count (43 -> 45) nor the file size, the two
candidates named in the revert. It is the size of a SINGLE `run:` step, and the
padding control proves it is size rather than anything in the content. Bracketed:
18,965 bytes resolves, 21,109 bytes does not. #161's axe step was 22,439.

FIX: the axe runner is now hydra-gates/scripts/axe-run.cjs — a real file, byte
identical to the one proven under #161 (md5 40ba28f6…) — checked out into the
Playwright job the same way the hydra-gates job already checks this repo out.
The axe step drops 22,439 -> 2,162 bytes, the largest `run:` step in the whole
file is now 8,533 bytes, and quality.yml is 201,108 bytes: SMALLER than the
207,576 it was before #161, because the program moved out of the YAML.

A 500-line Node program embedded in a YAML string was the real defect. Keeping
it in a file takes the entire class of failure off the table instead of leaving
the next comment one edit away from another fleet outage.

The behaviour is unchanged from #161 and re-verified against the shipped file:
openregister 2 serious/critical -> 0 scoped, every violation removed provably
core's; a violation injected INSIDE the container still reported at critical;
the same violation OUTSIDE not reported; a scope matching nothing aborts without
writing a report. Guard 4 asserts both halves on the live page every run.

Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
rubenvdlinde added a commit that referenced this pull request Aug 5, 2026
…ed runners — decidesk and doriath went red on gates 4, 24 and 33 (#173)

`hydra-gates-require-full-coverage` was flipped to default TRUE (#164) once the
runner could tell a not-applicable gate from a real gap. But this workflow is
consumed at @main while the gate package is consumed at the caller's own
`hydra-gates-ref`, so the new default was handed to runners that predate the
accounting entirely — and those count EVERY gate they did not run as a gap.

Measured, not inferred, both after the flip and both failing on nothing else:

  decidesk job 92432089644  ref v1.0.1  ALL 61 GATES GREEN
                            COVERAGE: 58 of 61 / DID NOT RUN: 4 24 33 -> exit 98
  doriath  job 92363547732  ref v1.0.1  same three gates, same verdict

Gates 4, 24 and 33 are the exact trio the product owner named as legitimately
not applicable. So the flip was producing the outcome it was made to prevent,
in every repo whose pin had not moved — and pins do not move on in-flight
branches.

The flag is now handed only to a package that can honour it. The probe reads the
package's own accounting (`_NA_GATES` in the runner, the `NOT APPLICABLE`
verdict in the wrapper) rather than comparing versions, because a pin may be a
tag, a branch, a SHA or a fork. Verified against every published tag:
v1.0.0/v1.0.1/v1.1.0/v1.2.0 -> withheld, v1.3.0/v1.4.0 -> applied. A checkout
missing either file is a broken checkout, not an old one, and fails.

Withholding is stated as a `::warning::` naming the pin and the fix. A control
that switches itself off quietly is a control that has stopped existing.

`--axe-enabled` is gated on the same probe. The old wrapper's parser ends in
`*) APP_DIR="$1"`, so an unrecognised flag is swallowed as a POSITIONAL and
clobbers the app dir set moments earlier: a v1.0.1 pin handed it aborts with
`FATAL: --app-dir '--axe-enabled' is not a directory. No gate ran.` — exit 99,
every gate lost, for a flag the caller never typed.

Same ref skew, second instance: axe-run.cjs landed in v1.4.0 and every caller is
pinned v1.3.0, so `enable-axe: true` would die on `Cannot find module` and then
present as "the axe report never arrived" — pointing at Playwright, the one
place the fault is not. Asserted at the checkout with a named error instead.

The two probe literals are a CONTRACT: renaming either would silently disable
coverage enforcement fleet-wide, with a warning blaming the caller's pin. The
package's own suite now asserts both, plus a reverse control proving the probe
does not match the pre-v1.3.0 shape of either file. Positive-controlled: with
both literals renamed in a copy, those assertions go red.

Four-way measurement on one fixture, one variable at a time:

  v1.0.1 + --require-full-coverage        exit 98  <- the observed failure
  v1.0.1, flag withheld (this change)     exit 0
  v1.4.0 + flag, only (a) gaps            exit 0   33 of 33 applicable gates ran
  v1.4.0 + flag + a structural gap        exit 98  [gate-24] SKIPPED (structural)

Also removes a stale comment left by the #161/#166/#168 cycle that instructed
future readers to embed the axe runner inline — the thing that made the whole
workflow unresolvable fleet-wide.

Suite 31 -> 34 tests, 0 failing. actionlint clean (positive-controlled: a
typo'd step id is reported). Largest `run:` step 8,533 B, far below the ~19 KB
that makes a workflow unresolvable.

Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
rubenvdlinde added a commit that referenced this pull request Aug 6, 2026
… on the failure mode that has no red (#177)

* feat(quality): stop pinning the gates package per repo, and gate main on the failure mode that has no red

The fleet pinned `hydra-gates-ref` in 23 repositories. A pin is a silent
expiry date, and it has now cost us twice:

  #159  22 repos sat on v1.0.1, predating a batch of gate fixes. 16 gates
        were dead fleet-wide and every one reported PASS — a gate whose
        helper never runs emits a tick identical to a real one.
  #173  the coverage flag was flipped to default true here, at @main, and
        reached v1.0.1 runners predating the accounting that makes it
        survivable. decidesk and doriath went red (exit 98) on gates 4, 24
        and 33 — the not-applicable gates that must never fail a run.

Both are one defect: this workflow is consumed at @main while the package
was consumed at a pin, so the two sides moved independently and a change on
one reached a runner from the other's past. The fix is no pin. The default
was already `main`; this makes that the documented contract and states the
rollback levers, because "always latest" is only safe with a way back:
revert on main for the whole fleet, or set the input explicitly for one
repo. A moving `v1` tag was considered and rejected — a tag nobody
re-points is indistinguishable from the stale pin being removed.

Unpinning means a broken main now reaches 23 repos at once, so main is
gated on the one failure mode that does not announce itself. An
unresolvable reusable workflow is not a red check: GitHub produces a run
with no jobs, or no run, and every dashboard stays green while every caller
quietly stops being checked. That is what a ~22 KB `run:` step did in #161,
reverted by #166 and re-landed by #168.

  quality-selftest.yml       an in-repo caller of ./.github/workflows/
                             quality.yml. It exists to be RESOLVED, not to
                             pass; its conclusion is not a verdict.
  quality-resolve-probe.yml  counts the JOBS that caller materialised.
                             `jobs == 0` is the outage signature, so the
                             count is the measurement and a conclusion is
                             deliberately not read. Split across two files
                             on purpose: with the call in the probe itself,
                             an unresolvable quality.yml would delete the
                             probe too and the required check would sit
                             PENDING FOREVER rather than go red.

Both guards are positive-controlled on every run. The probe first counts a
workflow that cannot exist and requires the answer to be 0 — a counter
returning a constant would otherwise pass forever. The lint is re-run at a
1-byte limit it cannot meet. The lint also reproduces the real outage:
against quality.yml at d68fb72 it names the 26959-byte block at line 2338.

hydra-gates-package.yml loses its `paths:` filter. It was defensible while
callers pinned; with the fleet on @main the blast radius of any merge here
is the whole fleet, and a required check with a paths filter leaves every
excluded PR pending forever.

Refs #159, #161, #166, #168, #173

* fix(selftest): widen the caller's permissions — a narrow grant is a second silent way to zero jobs

The selftest shipped with `contents: read`. Run 31073908079 came back
startup_failure with ZERO jobs, because quality.yml's journeydoc-capture job
declares `contents: write` + `actions: write` and permissions are validated
when the run is CREATED — `enable-journeydoc-capture: false` never gets a
chance to matter, since the job's `if:` is not reached.

This is a second route into the #161 outage class and it applies to every
caller in the fleet: a permission grant narrower than the shared workflow
requests produces no jobs and no red, indistinguishable from a healthy run
on any dashboard.

The probe caught it on its first execution, which is the evidence that it
works. Run 31073907988 recorded, in order:

  control count = 0
  OK — the counter can return 0, and reports it.
  quality-selftest.yml materialised 0 jobs at 7c3d160
  ##[error]quality.yml produced ZERO jobs. This is the #161 outage signature

So the guard has now been shown to fail on a REAL zero-jobs condition, not
only against its synthetic control.

---------

Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant