feat(quality): produce tests/axe/report.json so gate-33 (axe-core) can actually run

[rubenvdlinde]

The measured problem

hydra-gates gate-33 consumes tests/axe/report.json and fails on serious/critical axe violations. Measured 2026-08-03 across 13 repos: that file exists in no repository, and the scripts/run-browser-tests.sh documented as producing it exists in no app either. So gate-33 has never run anywhere, and every "all gates green" this fleet has produced excluded accessibility runtime checking — contrast, landmark structure, ARIA validity, live regions — entirely. v1.1.0 made the skip loud. This makes the report exist.

What is wired

Where it runs, and why. The playwright job is the only job in this file with both a booted Nextcloud and a browser, so axe runs there. hydra-gates has neither — it is a checkout, a shell script and python — so the report is handed over as an artifact. An artifact cannot be downloaded before its producer finishes, so hydra-gates now declares needs: [playwright].

!cancelled() on that job's if: is load-bearing: needs: implies success(), which would have silently deleted the gates job for every repo that has the gates on and Playwright off. The workflow's own verdict does not move — report already needed both jobs — only the position of the "Hydra Gates" check in the list.

The alternative (booting a second Nextcloud inside the gates job) duplicates ~150 lines for one gate, and the second instance would be a second thing that can differ from the one the specs ran against.

Opt-in, in the same shape as enable-hydra-gates. Three new inputs, all defaulting off/empty:

input	default
enable-axe	false	run axe and publish the report
axe-routes	""	routes to analyse; empty = the app's root route
axe-version	4.12.1	pinned, because the rule set decides the verdict

No repo is enabled by this change.

Default-off is not caution for its own sake: measured against a vanilla Nextcloud 34 with no Conduction app installed at all, /apps/files/ and /settings/user already carry three serious/critical violations (label, aria-input-field-name, aria-prohibited-attr). An app renders inside that shell, so the first run in any repo will be red and part of that red is core's.

Why the runner refuses to write a bad report

gate-33 reports PASS on a tests/axe/report.json containing exactly {}. Verified. A report written by a crashed step would therefore convert today's loud _skip into a silent false pass — strictly worse than where we started. So the runner writes nothing unless it earned it, behind three guards, and the gates job re-validates whatever arrives:

1. Positive control on every run. Before touching the app, axe is pointed at a scratch page carrying a deliberate button-name violation. If axe does not report it, axe cannot fail in this environment, so a subsequent empty violations array would be evidence about the harness. The run is abandoned.
2. HTTP status. A route answering >= 400 renders Nextcloud's error page, which axe reports as clean. Hard failure with the actionable message, not an empty result.
3. Non-empty result set. Zero passes AND zero violations is an absent measurement, not a clean one. (Stated in the code as a backstop: 4.12.1 actually throws on an uninjectable document, and that throw is caught and also writes nothing.)

Then in the gates job, the downloaded file is checked against the provenance the runner writes — testEngine.name, non-empty routesAnalysed with 2xx/3xx statuses and non-zero results, selfTest.detected — and deleted if it does not hold up. Deleting it restores the loud skip. It never invents a pass and never invents a failure.

Verification — both directions

Everything below was run against the copy extracted back out of this YAML, not the draft.

what	result
no report present	[gate-33] axe-core: SKIPPED — no tests/axe/report.json …, coverage 48 of 63, 33 named in GATES THAT DID NOT RUN
report with a deliberate icon-only button (button-name, critical) + a 1.6:1 paragraph (color-contrast, serious)	[gate-33] axe-core: FAIL — 2 serious/critical axe violation(s), exit code 2
the same page with those two defects removed	[gate-33] axe-core: PASS, coverage 49 of 63, 33 gone from the not-run list
live Nextcloud 34, logged in, 3 routes	FAIL — 3 serious/critical on real core violations; report 4.5 KB, real rule ids, 21/30/30 passes per route
report containing {}	gate-33 says PASS — this is the hazard the guards exist for; the validator rejects and deletes it

Each guard was also shown able to fire, writing no report in every case: self-test defeated (control button given a name) → refuses; route 404 → refuses; zero routes configured → refuses; unreachable host → refuses; a stale report from a previous attempt is removed up front, so "did not write" always means "no report".

The validator's eight branches were each exercised (good report / {} / truncated / zero routes / no self-test / route 404 / route with no results / no artifact at all) — every rejection deletes the file, every one prints why.

actionlint is clean on the file, and was positive-controlled: breaking steps.axe.outcome to a nonexistent step makes it exit 1 with that exact complaint, which also confirms the step id resolves. The gates package's own suites still pass (25/25 entry-point, 17 helper suites, 2 pre-existing quarantines).

Could not be verified here

This PR's own CI does not execute the new steps. quality.yml is a reusable workflow; ConductionNL/.github does not call it on itself, so nothing in this PR's check set boots a Nextcloud or runs axe. The compensating evidence is the live Nextcloud 34 run above, which exercised the same path end to end (login, three routes, real violations, artifact-shaped report). First real CI exercise will be the first repo to set enable-axe: true.

Scope

Workflow only. No repo enabled, no suppression baseline added or modified. The two doc touches (hydra-gates/README.md, the gate-33 comment block in run-hydra-gates.sh) point at the new input and record the {}-passes hazard; neither changes gate logic.