fix(gate-19): demote e2e-coverage to advisory while the fleet is repaired - #478
Merged
Conversation
gate-35 flags `alt=""` on a src whose expression carries a content noun.
That shape is usually the "I made gate-31 green by lying" tell, but it is
ALSO the shape of correct markup:
<a :href="item.link">
<img :src="item.thumbnailUrl" alt="">
<h4>{{ item.title }}</h4>
</a>
Here the link takes its accessible name from the heading, the image is
decorative, and `alt=""` is what WCAG H67 asks for -- giving the image its
own alt makes a screen reader announce the headline twice. There is no waiver
for gate-35, so the only ways out were to write a worse alt or to rename the
variable to dodge the noun list. Both are worse than the finding.
The noun test still decides WHICH images are suspicious. This adds a second
question: does the markup around it already name it? An image inside <a>,
<button>, <label> or <figure> is exempt only when that ancestor carries text
BESIDES the image.
Kept deliberately narrow, because widening is how this gate goes blind:
- only those four elements; a <div> wrapper names nothing and still fires
- the text must be INSIDE the ancestor -- a sibling link does not excuse an
image outside it
- a link whose only content IS the image still fires, because then the image
is the link's only possible name and alt="" leaves it nameless
Evidence: launchpad @ development, NewsWidget.vue lines 45 and 77 -- both
inside a link named by its own <h4>, both reported, no waiver available.
Verified:
- 46 unit tests pass; the 5 new exemption tests FAIL against the pre-fix
checker, so they bite rather than describe
- acceptance CLEAN arm: 0 findings, including the new NamedByContext.vue
whose 4 images the OLD checker flagged all 4 of
- acceptance PLANTED arm: all 10 findings still reported, including the new
NounsLinkOnly.vue anti-widening case
- test_gate_a11y_helper_wiring.sh 70/70, test_gate_a11y_markup_scope.sh ALL PASS
…ired gate-19 still runs, still counts, and still names every unannotated scenario. It no longer stops a merge. WHY Measured across the fleet on 2026-08-16: 1,173 unannotated scenarios in four apps alone — hermiq 491, docudesk 405, doriath 149, launchpad 128. On the one app taken end-to-end (nldesign) the honest split was roughly one scenario in five genuinely reachable from a browser; the rest were colour arithmetic, occ file writes, background-job intervals and generated email markup — real behaviour, proven by PHPUnit, just not by a page. Left blocking, that arithmetic has one cheap remedy and one expensive one, and the cheap one is mass '@e2e exclude'. nldesign was already carrying 585 of 786 scenarios excluded before any of this work started. A gate whose realistic remedy is 'waive most of it' has stopped measuring coverage. Worse, every app it reddens also hides that app's OTHER gates behind the same red job — gate-19 alone is why launchpad, doriath and docudesk cannot go green on findings they have already fixed. WHAT CHANGED - New _warn helper. NOT _skip: a skip means the gate did not judge, a warning means it judged, found something, and is deliberately not blocking. It still registers in _EMITTED_GATES so the COVERAGE line keeps counting gate-19 as having reported — demoting a gate must not quietly shrink the suite. - The summary prints the advisory count before the verdict and states plainly that green means 'nothing BLOCKING failed', not 'nothing was found'. The failure mode of a demotion is that the finding stops being read. - WARNING added to both verdict parsers (test_gate_acceptance_matrix.sh and gate_fixture_support.sh), kept in step as their comments require. TESTS The substance of every gate-19 assertion is unchanged — only the verdict word. prose-not-proof still requires the planted scenario to be SEEN and named, and its clean arm still requires PASS, so a demotion cannot be mistaken for the gate going blind. exit-code semantics still asserts the count 300 survives the exit byte uncorrupted. test_gate_exit_code_semantics.sh 9 passed, 0 failed test_gate_base_ref_delivery_channel.sh 12 passed, 0 failed This is a temporary demotion WITH AN OWNER, not a retirement — .github#477 tracks fixing gate-19 properly once the fleet is green.
rubenvdlinde
added a commit
that referenced
this pull request
Aug 16, 2026
… it missed, and bound _warn by name (#481) * fix(invariants): complete the #477 demotion in the two suites it missed, and bound it `.github@main`'s `Package invariants` job has been red since 13:15 today. Both failures come from #478 (`e1fe8a4`, gate-19 demoted to advisory), and NEITHER of them is the empty-scope property itself — that property still holds and is still asserted. WHICH PR, SETTLED BY RUNNING THE TESTS RATHER THAN READING CI #476's CI run was CANCELLED (concurrency `cancel-in-progress`, #478 pushed 7s later), so it was neither passed nor failed and could not be exonerated from CI. The two suites were run directly against each merge commit's tree instead: ff66c3b pre-#476 both suites GREEN 0b2c457 #476 merged (gate-35/25) both suites GREEN <- exonerated 530a849 #478 merged (gate-19) both suites RED, 1 assertion each 6f756a8 main (#479 merged) both suites RED, same two assertions WHAT EACH ASSERTION ASSERTED, AND WHAT IT GOT 1. test_gate_empty_scope_never_passes.sh, ARM 1 asserted: `[gate-19] ...: FAIL` over a fixture carrying one planted unannotated scenario. got: `[gate-19] e2e-coverage: WARNING — 1 scenario(s) missing @e2e`. ARM 1 is the POSITIVE CONTROL for the fixture — everything after it is meaningless unless the gate demonstrably sees the planted defect. #477 deliberately moved gate-19's verdict word, not its detection. The arm matched the word rather than the property. 2. test_gate_discarded_counts_and_empty_deltas.sh, ARM P3 asserted: the literal `(PASS|FAIL|NOT APPLICABLE|SKIPPED)` appears in BOTH gate_fixture_support.sh and test_gate_acceptance_matrix.sh. got: both files now read `(PASS|FAIL|WARNING|NOT APPLICABLE|SKIPPED)` — changed IN STEP by #478, exactly as their comments require. The stated property is "the two parsers must stay in step". The implemented mechanism froze the vocabulary instead, so a correct in-step change failed it, and its own repair path was to retype a literal nobody can check against the files it polices. WHICH SIDE WAS WRONG The tests, on both counts — but only in mechanism, and the repair strengthens rather than relaxes: - ARM 1 now accepts FAIL *or* WARNING for gate-19 AND requires the line to name a non-zero count, because a demotion's real failure mode is the finding ceasing to be read. gate-25 is not demoted and keeps the strict FAIL. PASS and NOT APPLICABLE are still refused, and ARM 2 still requires NOT APPLICABLE over the empty scope, so the two remain distinguishable. - A new assertion requires the run SUMMARY to announce the advisory findings and to say that green means "nothing BLOCKING failed". - ARM P3 now extracts the alternation from each parser and requires them to be byte-identical, with the four core verdict forms as a floor so "agreement" cannot be reached by gutting both. - NEW ARM P4: the set of gates allowed to call `_warn` is an explicit allowlist (`19`, .github#477). `_warn` is otherwise one line away from being a soft-fail switch for any gate in the suite, and nothing in this package would have noticed a second one. This does not stop a demotion; it stops a silent one. The empty-scope property is untouched. ARM 2 (`NOT APPLICABLE`, never PASS) and ARM 6 (`no gate reports PASS over a diff that excludes every planted defect`, positive control: 18 gates FAIL the planted tree at full scope) pass unchanged before and after. POSITIVE CONTROLS — each mutation run against an unmodified copy of the package gate-19 `_warn` -> `_pass 19` ARM 1 FAILS: "got: PASS" gate-19 warning with the count removed ARM 1 FAILS: "named no non-zero count" WARNING added to ONE parser only ARM P3 FAILS, naming both alternations gate-18's advisory turned into `_warn` ARM P4 FAILS: "found '18 19', allowed '19'" Both suites also carry the drift/allowlist controls inline, so a future extractor that returns empty for everything cannot call that agreement. The assertion floor in the discarded-counts suite moves 34 -> 37 to match. NOT DONE, DELIBERATELY #478's demotion of gate-19 is a fleet policy decision with a stated reason and a tracking issue; reverting it is the coordinator's call, not this PR's. gate-18's advisory line is printed AFTER its verdict, so `head -1` parsers are safe today — but it is now shape-matchable as a verdict, and that safety is an ordering rather than a mechanism. * fix(invariants): the THIRD verdict parser was never taught WARNING, and it fails silently There are three verdict parsers in this package, not two. #477 taught `gf_verdict` and the acceptance matrix the new WARNING form. It left `_verdict_set` in test_gate_base_ref_delivery_channel.sh — the one that decides "every gate returned the same verdict through both channels" — matching only (PASS|FAIL|NOT APPLICABLE|SKIPPED). Nothing went red, and that is the defect. A verdict word this pattern does not know is DROPPED, not flagged. So gate-19 fell out of both channels' verdict sets, the comparison went on matching over a set that no longer contained the gate that suite exists to watch, and it reported 12 passed / 0 failed. MEASURED, on the real fixture: main @ 6f756a8 "every gate returned the same verdict ... (69 gate(s) compared)" this branch "every gate returned the same verdict ... (70 gate(s) compared)" POSITIVE CONTROL — an A/B against a REAL leak, not an argument. A copy of the runner was made to inflate gate-19's count by one when the base arrived through the environment rather than through --base, i.e. exactly the .github#416 defect this suite was built to catch, planted in gate-19: suite as on main today PASS — 12 passed, 0 failed, "(69 gate(s) compared)" suite with this commit FAIL — "THE DELIVERY CHANNEL CHANGED THE VERDICT" The main-today arm prints both arms' differing counts (`WARNING — 1` vs `WARNING — 2`) two lines above the assertion that then declares them identical. Both this and the ordinary case are now mechanised: ARM P3 of test_gate_discarded_counts_and_empty_deltas.sh compares ALL THREE parsers by name and requires one shared alternation, so a fourth parser or a fourth verdict word cannot be added to some of them. --------- Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gate-19 still runs, still counts, and still names every unannotated scenario.
It no longer stops a merge. Tracked for proper repair in #477.
Why
Measured across
developmenton 2026-08-16 — 1,173 unannotated scenariosin four apps alone (hermiq 491, docudesk 405, doriath 149, launchpad 128).
On nldesign, the one app taken end-to-end, roughly one scenario in five was
genuinely reachable from a browser. The rest were colour arithmetic,
occfilewrites,
TimedJobintervals and generated email markup — real behaviour,proven by PHPUnit, just not by a page.
Left blocking, that arithmetic has one cheap remedy and one expensive one, and
the cheap one is mass
@e2e exclude. nldesign was already carrying 585 of786 scenarios excluded before this work began. A gate whose realistic remedy
is "waive most of it" has stopped measuring coverage.
There is also a structural cost: gate-19 is a shared red. Every app it fails
also fails
Hydra GatesandQuality Report, hiding that app's other findingsbehind the same red job. launchpad, doriath and docudesk each have gate work
already done that cannot show green while gate-19 blocks.
What changed
_warnhelper. Deliberately not_skip: a skip means the gate didnot judge; a warning means it judged, found something, and is not blocking.
It still registers in
_EMITTED_GATES, so the COVERAGE line keeps countinggate-19 as having reported — demoting a gate must not quietly shrink the
suite.
and states plainly that green means "nothing blocking failed", not "nothing
was found". The failure mode of a demotion is the finding silently ceasing
to be read.
WARNINGadded to both verdict parsers (test_gate_acceptance_matrix.shand
gate_fixture_support.sh), kept in step as their own comments require —without this a demoted gate reads as "emitted NO verdict line at all".
Tests
The substance of every gate-19 assertion is unchanged; only the verdict word
moved.
prose-not-proofstill requires the planted scenario to be seen and named,and its clean arm still requires
PASS— so a demotion cannot be mistakenfor the gate going blind.
uncorrupted; that property is about the count, not the severity.
test_gate_exit_code_semantics.shtest_gate_base_ref_delivery_channel.shProperty 5 of the exit-code suite now reports wrapper exit
0over a fixturewith 300 uncovered scenarios — which is the demotion working, verified rather
than assumed.
Stacking
Opened on top of
fix/gate-35-named-by-context(#476). Once that merges, thisdiff is gate-19 only.