fix(quality): scope axe to the app's own DOM, and prove the scope is not a mute by rubenvdlinde · Pull Request #161 · ConductionNL/.github

rubenvdlinde · 2026-08-05T13:02:46Z

What was wrong

axe analysed the whole document. Every serious/critical violation in Nextcloud's chrome was therefore attributed to whichever app was under test.

Measured on a live Nextcloud 34, unscoped:

route	serious/critical nodes	whose?
`/apps/openregister/`	2	`color-contrast` @ `.unified-search-input__label` (core's unified search, inside `#header`) + `role-img-alt` @ `:root`
`/apps/hermiq/`	1	`color-contrast` @ `.unified-search-input__label`

Every one of them is core's. Scoped to #content-vue, #content, both routes go to 0. A control that fires on defects the app cannot fix gets switched off — that is the failure mode this closes.

The selector was measured, not guessed

Every candidate that has been proposed, probed on a live NC 34.0.2 while logged in:

route	`#content-vue`	`#content`	`#app-content-vue`	`main`	`#app-content`	`main#content`
`/apps/files/`	yes	NO	yes	yes	NO	NO
`/settings/user`	yes	yes	yes	yes	NO	NO
`/apps/dashboard/`	NO	yes	NO	yes	NO	NO
`/apps/openregister/`	yes	yes	yes	yes	NO	NO
`/apps/hermiq/`	yes	yes	yes	yes	NO	NO

#app-content and main#content do not exist on NC34 at all — either would match nothing and abort the run.
No single selector covers every page, hence the two-selector default #content-vue, #content.
#app-content-vue/main also cover every page but sit inside #content-vue and exclude the app navigation — which a Conduction app renders itself. Scoping to them would hide real app defects. That is muting, and it is deliberately not the default.

Three-way proof — the third is the one that distinguishes scoping from muting

Scoping and muting look identical from outside: both produce fewer violations. So guard 4 asserts both halves on the live page, on every run, and refuses to write a report if either fails.

Verified against the copy extracted back out of this YAML (not the draft), against /apps/openregister/:

Scoped baseline — 2 -> 0 serious/critical; every removed node provably core's (set-diff of full node lists, not totals).
Violation injected INSIDE the container — <button></button> appended to #content-vue is still reported as button-name / critical. insideProbeReported: true.
The same violation injected OUTSIDE (#header / document.body) — not reported when scoped; reported by an unscoped control run on the same injected page, proving the injection was real and detectable and that the scope is what excluded it. outsideProbeReported: false.

The impact filter is untouched — still serious/critical only. Nothing was weakened to reach a green.

Dead-gate guards (an absent result must never read as a clean one)

Four failure modes verified to abort without writing a report, so gate-33 skips loudly instead:

condition	measured behaviour
scope matches nothing (`#app-content, main#content`)	named die, exit 2, no report written
`axe-include-selector` empty	named die, exit 2, no report — an empty scope is not "scope to everything"
unparseable selector	named die naming the selector and the parse error
every selector counts 0 on a route	named die before axe is even called

Measured asymmetry, documented in the input: an include that matches nothing throws; an exclude that matches nothing is silently ignored. So a typo in exclude fails open (cannot hide anything) but does quietly stop excluding.

axe-include-selector: body opts out of scoping. Guard 4's outside-probe half then cannot execute — that is recorded as outsideProbeExecuted: false and announced as a ::warning::, because a control that did not run must never look like one that passed.

The report now carries scope and scopeControl as provenance: two reports with the same empty violations array mean opposite things depending on what was looked at.

Honest limit

On vanilla Nextcloud's own routes (/apps/files/, /settings/user) scoping removes nothing — 3 serious/critical before and after. Those defects (label, aria-input-field-name, aria-prohibited-attr) live in the Files and Settings apps' own content, not in the chrome — verified by walking each violating node's ancestry: all three are inside #app-content-vue. #header is clean at serious/critical.

So "a vanilla page reports 0 once scoped" is not achievable by scoping, and achieving it would require excluding real app content — i.e. muting. What is achieved, and is what the gate actually measures (axe-routes defaults to the app's own root route): an app's own page reports 0 once scoped, and everything removed is provably core's.

axe-exclude-selector exists for a genuinely-embedded third-party region and stays empty by default; it is documented as the one lever here that can hide a real defect.

Verification

actionlint positive-controlled first (exit 1 with 2 real findings on a deliberately broken workflow — an earlier attempt with an invalid flag returned exit 2 uniformly for both arms, which is an instrument error, not a verdict). Base origin/main exit 0, this branch exit 0.
Runner re-extracted from the committed YAML and re-run end-to-end after the last edit.
Branched from current main (8c58428); feat(quality): produce tests/axe/report.json so gate-33 (axe-core) can actually run #148 and fix(quality): enable-axe went GREEN having measured nothing; the needs-deletion suspicion was a cancelled run #151 both intact.

🤖 Generated with Claude Code

…not a mute axe analysed the whole document, so every serious/critical violation in Nextcloud's chrome was attributed to whichever app happened to be under test. Measured on a live instance: /apps/openregister/ reported 2 serious/critical nodes and /apps/hermiq/ 1, and ALL of them were core's — `color-contrast` on `.unified-search-input__label` (Nextcloud's unified search, inside #header) and `role-img-alt` on `:root`. Scoped, both go to 0. A control that fires on defects the app cannot fix gets switched off, which is the failure mode this closes. The selector was measured, not guessed. On Nextcloud 34.0.2, `#app-content` and `main#content` do not exist on any page, and no single selector covers all of them: `#content-vue` is absent on the dashboard, `#content` is absent on /apps/files/. Hence the two-selector default `#content-vue, #content`. `#app-content-vue`/`main` also cover every page but sit inside `#content-vue` and exclude the app's navigation — which a Conduction app renders itself — so scoping to them would hide real app defects. Scoping and muting look identical from the outside: both produce fewer violations. So guard 4 asserts both halves on the live page every run — a deliberate `button-name` injected INSIDE the scope must still be reported, and the same violation injected OUTSIDE must not be. Verified against the copy extracted back out of this YAML: inside probe reported, outside probe not, and when the scope is `body` the outside half cannot execute and says so rather than passing silently. Four failure modes verified to abort without writing a report, so gate-33 skips loudly instead of reading an absent result as a clean one: a scope matching nothing (axe itself throws), an empty scope, an unparseable selector, and a route where every selector counts 0. Honest limit: on vanilla Nextcloud's OWN routes (/apps/files/, /settings/user) scoping removes nothing — 3 serious/critical before and after — because those defects live in the Files and Settings apps' content, not in the chrome. Removing them would require excluding real app content. axe-exclude-selector exists for that and stays empty by default.

…cope is not a mute (#161)" (#166) This reverts commit d68fb72. Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

…22 KB `run:` step is what made the workflow unresolvable (#168) #161 scoped axe to the app's own DOM and was reverted in #166 because every caller in the fleet started up with ZERO jobs and a run `name` equal to the raw file path — GitHub never RESOLVED quality.yml. The revert was right. This re-lands the same measured behaviour in a shape GitHub resolves, and names the cause rather than guessing at it. CAUSE, bisected with a controlled probe (a `uses: ./…` caller inside this repo, so resolution can be tested on a branch without pointing an app repo at unmerged code). Read the JOB COUNT, never the conclusion — `failure` with jobs>0 means it resolved and something inside failed, which is a PASS for the probe: baseline (current main) -> 25 jobs RESOLVES #161 re-applied in full -> 0 jobs outage reproduced 45 workflow_call inputs, step untouched -> 25 jobs RESOLVES whole FILE padded to 226 KB, step untouched -> 25 jobs RESOLVES ONE `run:` step padded to 21 KB with INERT JS COMMENTS, nothing else changed -> 0 jobs BREAKS So it is neither the input count (43 -> 45) nor the file size, the two candidates named in the revert. It is the size of a SINGLE `run:` step, and the padding control proves it is size rather than anything in the content. Bracketed: 18,965 bytes resolves, 21,109 bytes does not. #161's axe step was 22,439. FIX: the axe runner is now hydra-gates/scripts/axe-run.cjs — a real file, byte identical to the one proven under #161 (md5 40ba28f6…) — checked out into the Playwright job the same way the hydra-gates job already checks this repo out. The axe step drops 22,439 -> 2,162 bytes, the largest `run:` step in the whole file is now 8,533 bytes, and quality.yml is 201,108 bytes: SMALLER than the 207,576 it was before #161, because the program moved out of the YAML. A 500-line Node program embedded in a YAML string was the real defect. Keeping it in a file takes the entire class of failure off the table instead of leaving the next comment one edit away from another fleet outage. The behaviour is unchanged from #161 and re-verified against the shipped file: openregister 2 serious/critical -> 0 scoped, every violation removed provably core's; a violation injected INSIDE the container still reported at critical; the same violation OUTSIDE not reported; a scope matching nothing aborts without writing a report. Guard 4 asserts both halves on the live page every run. Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

@main

…ed runners — decidesk and doriath went red on gates 4, 24 and 33 (#173) `hydra-gates-require-full-coverage` was flipped to default TRUE (#164) once the runner could tell a not-applicable gate from a real gap. But this workflow is consumed at @main while the gate package is consumed at the caller's own `hydra-gates-ref`, so the new default was handed to runners that predate the accounting entirely — and those count EVERY gate they did not run as a gap. Measured, not inferred, both after the flip and both failing on nothing else: decidesk job 92432089644 ref v1.0.1 ALL 61 GATES GREEN COVERAGE: 58 of 61 / DID NOT RUN: 4 24 33 -> exit 98 doriath job 92363547732 ref v1.0.1 same three gates, same verdict Gates 4, 24 and 33 are the exact trio the product owner named as legitimately not applicable. So the flip was producing the outcome it was made to prevent, in every repo whose pin had not moved — and pins do not move on in-flight branches. The flag is now handed only to a package that can honour it. The probe reads the package's own accounting (`_NA_GATES` in the runner, the `NOT APPLICABLE` verdict in the wrapper) rather than comparing versions, because a pin may be a tag, a branch, a SHA or a fork. Verified against every published tag: v1.0.0/v1.0.1/v1.1.0/v1.2.0 -> withheld, v1.3.0/v1.4.0 -> applied. A checkout missing either file is a broken checkout, not an old one, and fails. Withholding is stated as a `::warning::` naming the pin and the fix. A control that switches itself off quietly is a control that has stopped existing. `--axe-enabled` is gated on the same probe. The old wrapper's parser ends in `*) APP_DIR="$1"`, so an unrecognised flag is swallowed as a POSITIONAL and clobbers the app dir set moments earlier: a v1.0.1 pin handed it aborts with `FATAL: --app-dir '--axe-enabled' is not a directory. No gate ran.` — exit 99, every gate lost, for a flag the caller never typed. Same ref skew, second instance: axe-run.cjs landed in v1.4.0 and every caller is pinned v1.3.0, so `enable-axe: true` would die on `Cannot find module` and then present as "the axe report never arrived" — pointing at Playwright, the one place the fault is not. Asserted at the checkout with a named error instead. The two probe literals are a CONTRACT: renaming either would silently disable coverage enforcement fleet-wide, with a warning blaming the caller's pin. The package's own suite now asserts both, plus a reverse control proving the probe does not match the pre-v1.3.0 shape of either file. Positive-controlled: with both literals renamed in a copy, those assertions go red. Four-way measurement on one fixture, one variable at a time: v1.0.1 + --require-full-coverage exit 98 <- the observed failure v1.0.1, flag withheld (this change) exit 0 v1.4.0 + flag, only (a) gaps exit 0 33 of 33 applicable gates ran v1.4.0 + flag + a structural gap exit 98 [gate-24] SKIPPED (structural) Also removes a stale comment left by the #161/#166/#168 cycle that instructed future readers to embed the axe runner inline — the thing that made the whole workflow unresolvable fleet-wide. Suite 31 -> 34 tests, 0 failing. actionlint clean (positive-controlled: a typo'd step id is reported). Largest `run:` step 8,533 B, far below the ~19 KB that makes a workflow unresolvable. Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

…econd silent way to zero jobs The selftest shipped with `contents: read`. Run 31073908079 came back startup_failure with ZERO jobs, because quality.yml's journeydoc-capture job declares `contents: write` + `actions: write` and permissions are validated when the run is CREATED — `enable-journeydoc-capture: false` never gets a chance to matter, since the job's `if:` is not reached. This is a second route into the #161 outage class and it applies to every caller in the fleet: a permission grant narrower than the shared workflow requests produces no jobs and no red, indistinguishable from a healthy run on any dashboard. The probe caught it on its first execution, which is the evidence that it works. Run 31073907988 recorded, in order: control count = 0 OK — the counter can return 0, and reports it. quality-selftest.yml materialised 0 jobs at 7c3d160 ##[error]quality.yml produced ZERO jobs. This is the #161 outage signature So the guard has now been shown to fail on a REAL zero-jobs condition, not only against its synthetic control.

@main

… on the failure mode that has no red (#177) * feat(quality): stop pinning the gates package per repo, and gate main on the failure mode that has no red The fleet pinned `hydra-gates-ref` in 23 repositories. A pin is a silent expiry date, and it has now cost us twice: #159 22 repos sat on v1.0.1, predating a batch of gate fixes. 16 gates were dead fleet-wide and every one reported PASS — a gate whose helper never runs emits a tick identical to a real one. #173 the coverage flag was flipped to default true here, at @main, and reached v1.0.1 runners predating the accounting that makes it survivable. decidesk and doriath went red (exit 98) on gates 4, 24 and 33 — the not-applicable gates that must never fail a run. Both are one defect: this workflow is consumed at @main while the package was consumed at a pin, so the two sides moved independently and a change on one reached a runner from the other's past. The fix is no pin. The default was already `main`; this makes that the documented contract and states the rollback levers, because "always latest" is only safe with a way back: revert on main for the whole fleet, or set the input explicitly for one repo. A moving `v1` tag was considered and rejected — a tag nobody re-points is indistinguishable from the stale pin being removed. Unpinning means a broken main now reaches 23 repos at once, so main is gated on the one failure mode that does not announce itself. An unresolvable reusable workflow is not a red check: GitHub produces a run with no jobs, or no run, and every dashboard stays green while every caller quietly stops being checked. That is what a ~22 KB `run:` step did in #161, reverted by #166 and re-landed by #168. quality-selftest.yml an in-repo caller of ./.github/workflows/ quality.yml. It exists to be RESOLVED, not to pass; its conclusion is not a verdict. quality-resolve-probe.yml counts the JOBS that caller materialised. `jobs == 0` is the outage signature, so the count is the measurement and a conclusion is deliberately not read. Split across two files on purpose: with the call in the probe itself, an unresolvable quality.yml would delete the probe too and the required check would sit PENDING FOREVER rather than go red. Both guards are positive-controlled on every run. The probe first counts a workflow that cannot exist and requires the answer to be 0 — a counter returning a constant would otherwise pass forever. The lint is re-run at a 1-byte limit it cannot meet. The lint also reproduces the real outage: against quality.yml at d68fb72 it names the 26959-byte block at line 2338. hydra-gates-package.yml loses its `paths:` filter. It was defensible while callers pinned; with the fleet on @main the blast radius of any merge here is the whole fleet, and a required check with a paths filter leaves every excluded PR pending forever. Refs #159, #161, #166, #168, #173 * fix(selftest): widen the caller's permissions — a narrow grant is a second silent way to zero jobs The selftest shipped with `contents: read`. Run 31073908079 came back startup_failure with ZERO jobs, because quality.yml's journeydoc-capture job declares `contents: write` + `actions: write` and permissions are validated when the run is CREATED — `enable-journeydoc-capture: false` never gets a chance to matter, since the job's `if:` is not reached. This is a second route into the #161 outage class and it applies to every caller in the fleet: a permission grant narrower than the shared workflow requests produces no jobs and no red, indistinguishable from a healthy run on any dashboard. The probe caught it on its first execution, which is the evidence that it works. Run 31073907988 recorded, in order: control count = 0 OK — the counter can return 0, and reports it. quality-selftest.yml materialised 0 jobs at 7c3d160 ##[error]quality.yml produced ZERO jobs. This is the #161 outage signature So the guard has now been shown to fail on a REAL zero-jobs condition, not only against its synthetic control. --------- Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

The opt-out said axe was off because "a vanilla Nextcloud 34 already carries serious/critical violations on core's OWN routes that DOM scoping does not remove". That was true when written. ConductionNL/.github#161 — "scope axe to the app's own DOM, and prove the scope is not a mute", re-landed as #168 after a 22 KB inline `run:` step made the workflow unresolvable — gave the runner an AxeBuilder.include scope and measured /apps/openregister/ from 2 serious/critical violations to 0 and /apps/hermiq/ from 1 to 0, every one of them core's. The exclusion this opt-out was hand-rolling now exists upstream, so the stated reason no longer holds. Turned on with an explicit include selector rather than the `#content-vue, #content` default. DocuDesk mounts on a dedicated host element: templates/index.php renders exactly `<div id="docudesk-app"></div>` and src/main.js mounts there, for a documented reason — Vue 3's mount() renders INSIDE the match, so mounting on `#content` would nest the app inside Nextcloud's own wrapper from layout.user.php and leave two `#content` elements with which one matched undefined. Core's layout wraps our div in `#content`, so the DEFAULT selector would match and drag core's chrome — navigation, header, app menu — back into scope, re-attributing core's violations to this app. `#docudesk-app` is this app's own rendered DOM and nothing else. No baseline accompanies this, deliberately. The runner already proves the scope is not a mute on every run: it injects a violation INSIDE the container and asserts axe reports it, then injects the same violation OUTSIDE and asserts axe does not, and a selector matching nothing is a hard failure rather than an empty pass. A rule/fingerprint baseline would also be the wrong instrument — DOM scoping excludes TERRITORY, a baseline suppresses FINDINGS, including future ones on our own markup. This is an opt-in to ENFORCEMENT: if no usable report reaches the hydra-gates job the job now fails with a named reason instead of going green with gate-33 silently SKIPPED, which is what it has done here since the gate was written. The first run is the measurement.

gate-33 (axe-core) has reported NOT APPLICABLE on every run this repo has ever made, because `enable-axe` was never set. That is not a pass: runtime accessibility — contrast, landmarks, ARIA validity, live regions — has never been measured here at all, by a choice recorded in this file. The recorded reason was "a vanilla Nextcloud 34 already carries serious/critical violations from core's own UI". An exemption's reason is a testable claim, and this one names a STATE OF THE WORLD rather than a test artifact — so it rots. It has: 1. ConductionNL/.github#161 (re-landed as #168) gave the axe runner an AxeBuilder.include SCOPE, so core's chrome is no longer in the measurement. It is a scope, not a mute, and the runner proves that on every run — it injects a deliberate violation INSIDE the container and asserts axe reports it, then injects the same violation OUTSIDE and asserts axe does not. A selector matching nothing is a hard failure, not an empty pass. 2. ConductionNL/.github#338 fixed an axe runner that had NEVER resolved @playwright/test in any repo — so nobody could have enabled this before tonight even if they had wanted to. Measured rather than inherited: docudesk runs with `enable-axe: true` today and its gate run reports `[gate-33] axe-core: PASS` (run 31461514843). The blocker is gone, so the excuse does not hold here either. `axe-include-selector: '#zaakafhandelapp-app'` — NOT the `#content-vue, #content` default. templates/index.php:34 renders exactly `<div id="zaakafhandelapp-app">` and src/main.js:151 mounts there, because Vue 3's mount() renders INSIDE the match and mounting on `#content` would nest the app inside Nextcloud's own wrapper. Core's layout wraps our div in `#content`, so the DEFAULT selector would match and drag core's navigation, header and app menu back into scope — re-attributing core's violations to this app, which is exactly what the old opt-out was worried about. This selector is this app's own rendered DOM and nothing else. No baseline is taken with this. DOM scoping excludes TERRITORY; a baseline suppresses FINDINGS, including future ones on our own markup. If this run reports violations in our own DOM they are ours, and the fix is the markup.

rubenvdlinde merged commit d68fb72 into main Aug 5, 2026
4 checks passed

rubenvdlinde deleted the fix/axe-scope-to-app-dom branch August 5, 2026 13:29

rubenvdlinde added a commit that referenced this pull request Aug 5, 2026

Revert "fix(quality): scope axe to the app's own DOM, and prove the s…

25e28fa

…cope is not a mute (#161)" (#166) This reverts commit d68fb72. Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>

rubenvdlinde mentioned this pull request Aug 5, 2026

Re-land #161 (axe scoping) in a form GitHub can resolve — and add a startup-failure canary #167

Open

rubenvdlinde pushed a commit that referenced this pull request Aug 5, 2026

test(ci): NEGATIVE CONTROL — re-apply #161 in full, expect zero jobs

248bae3

rubenvdlinde pushed a commit that referenced this pull request Aug 5, 2026

test(ci): BISECT D — pad ONLY the axe run: step to #161's byte size

474848a

rubenvdlinde mentioned this pull request Aug 5, 2026

fix(quality): re-land axe DOM scoping, with the runner in a file — a 22 KB run: step is what made the workflow unresolvable #168

Merged

rubenvdlinde mentioned this pull request Aug 5, 2026

fix(quality): the coverage flag went ON at @main and reached OLD pinned runners — decidesk and doriath went red on gates 4, 24 and 33 #173

Merged

rubenvdlinde mentioned this pull request Aug 6, 2026

feat(quality): stop pinning the gates package per repo, and gate main on the failure mode that has no red #177

Merged

rubenvdlinde mentioned this pull request Aug 6, 2026

fix(quality): measure the coverage ratchet floor at the merge base, and delete the staleness job #181

Merged

rubenvdlinde mentioned this pull request Aug 11, 2026

ci: enable axe, scoped to this app's own DOM (gate-33: NA → measured) ConductionNL/zaakafhandelapp#343

Merged

Navigation Menu

Uh oh!

fix(quality): scope axe to the app's own DOM, and prove the scope is not a mute - #161

fix(quality): scope axe to the app's own DOM, and prove the scope is not a mute#161
rubenvdlinde merged 1 commit into
mainfrom
fix/axe-scope-to-app-dom

rubenvdlinde commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rubenvdlinde commented Aug 5, 2026

What was wrong

The selector was measured, not guessed

Three-way proof — the third is the one that distinguishes scoping from muting

Dead-gate guards (an absent result must never read as a clean one)

Honest limit

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant