Skip to content

test(gates): close the regression gap — 58 of 65 gates now fixtured, and the two registries reconciled against the code - #405

Merged
rubenvdlinde merged 3 commits into
mainfrom
fixtures/uncovered-gate-bundles
Aug 12, 2026
Merged

test(gates): close the regression gap — 58 of 65 gates now fixtured, and the two registries reconciled against the code#405
rubenvdlinde merged 3 commits into
mainfrom
fixtures/uncovered-gate-bundles

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

What this closes

The acceptance corpus protected 22 of 65 gates by the ratchet's own arithmetic.
The other 43 sat in UNCOVERED.md, and thirty-six of those rows were wrong
nine because nothing blocked authoring a bundle, twenty-seven because a planted
defect was already being driven through the real wrapper by a suite under
scripts/lib/, which neither registry knew about.

After this PR the ratchet reports 58 of 65, and the seven that remain each
name a specific thing a fixture directory cannot supply.

1. Five new bundles — nine gates, security and data first

bundle gates what the planted arm is
semantic-authz/ 9 semantic-auth, 6 orphan-auth #[NoAdminRequired] over a requireAdmin() body; an uncalled isTransitionAllowed(string $userId, …)
apphost-prelude/ 64 apphost-autoload-prelude an OCA\OpenRegister\AppHost\Bootstrap reference in register() with no ADR-040 prelude
register-dialect/ 18 notification-dialect, 51 schema-property-titles a lifecycleEnter trigger and a rule-level singular channel; a property with no title, and one with a title and no description
dashboard-and-detail/ 15 dashboard-antipattern, 55 detail-page-discipline <CnDashboardPage> inside a #widget-kpis slot, and a type:"custom" page used as a widget; page-level widgets[] shadowing config.widgets, and a deprecated summaryAggregates
manifest-planes/ 62 store-plane, 63 settings-surface a Store entry rendering ViewGridOutline and a Catalog over catalog_item; adminSettings[], a reserved-name settings page, and a Settings > Settings foldout

Thirteen expect.conf rows across nine gates. Every gate that could carry two
mechanisms carries two rows, each naming a subject in a different file, rule or
page
— because the driver grades a verdict plus ONE named subject, so a
one-row bundle stays green over any regression that leaves one finding standing.
That is the method=preamble / method=preambleForbiddenCode lesson applied
before it could bite.

Both arms of every bundle were run through scripts/run-hydra-gates.sh by
test_gate_acceptance_matrix.sh: 102 assertions, 0 failures on the fixture
arms.

2. Every bundle is revert-proven, one mechanism at a time

Each row was proven by BREAKING THE GATE and watching the named subject leave
the log. A prediction was written before each run; all nine matched.

gate mutation predicted measured
15 disable the slot rule widget=kpis gone, page=metrics survives exactly that, # count=1
55 disable rule (a) thingDetail gone, caseDetail survives exactly that
18 disable the singular-channel test token=channel gone, token=lifecycleEnter survives exactly that
51 accept a property that carries a title closedAt gone, governanceBody survives exactly that
62 disable the Tier A icon rule row 1 gone, one glyph, two meanings + Catalog survive exactly that, 3 → 2
63 disable the adminSettings[] rule row 1 gone, other two survive exactly that, 3 → 2
64 disable BOOTSTRAP_REF no finding at all apphost-autoload-prelude: OK
9 disable the admin-body test method=adminPanelState gone no output
6 widen: drop the auth-signal requirement 1 → 2 findings, so FAIL — 1 orphan method(s) no longer matches isValidCode joined it

The gate-6 arm is the anti-widening direction on purpose: its near-miss lives in
the same file as its true positive, so the row is two-directional even though it
grades a count (see below).

3. The reconciliation — the deliverable, not a side effect

UNCOVERED.md loses thirty-six rows. Nine to the new bundles; twenty-seven to
COVERED-ELSEWHERE.md, each row naming the suite and quoting the assertion
that proves the claim
, every quote observed in a run of that suite today —
test_gate_1_11_honest_verdicts.sh (1, 3, 5, 8, 10, 11),
test_gate_a11y_markup_scope.sh (34, 37, 39, 40, 42, 43, 44, 45),
test_gate_5661_empty_scope_is_not_a_pass.sh (56–60),
test_gate_13_multiline_dialog.sh (13), test_gate_route_auth.sh +
test_gate_fq_route_names.sh (5, 14), test_gate_or_objectservice_surface.sh +
test_gate_20_comment_masking.sh (17, 20), test_gate_46_tests_scope.sh (46),
test_gate_monitoring_and_skiplink.sh (38), test_gate_45_to_55_acceptance.sh
(50, 53).

Three gates were deliberately NOT moved. 12, 41 and 52 have a planted arm
but no anti-widening arm anywhere, so COVERED-ELSEWHERE's "planted and
clean" claim would be an overclaim. They get a new partial-elsewhere category
that names the suite, the assertion that exists, and the missing half.

gate-22's reason had rotted and is corrected. It said "no node_modules
ships anywhere in this package, so the clean arm cannot go green".
.github/workflows/hydra-gates-package.yml installs ajv before the helper-suite
step, and a schema-valid manifest was measured clean today. What actually blocks
a bundle is that the acceptance driver has no ajv preflight, so a wiring fault
would be reported as a gate defect — the exact false pass
test_gate_45_to_55_acceptance.sh grew its own preflight to prevent. A reason
naming a state of the world rots; a reason naming a test artefact does not.

4. Two things found while doing this

gate-6 is the only gate in the runner whose findings log is a mktemp path.
hydra-gate-orphan-auth.XXXXXX.log, chosen for hydra#110 so parallel app runs
cannot clobber counts. The acceptance driver resolves <log-basename> as a
literal filename, so gate-6 cannot be graded by subject at all — its row
uses the - (stdout) form and grades the count. That is weaker and it is
written into the bundle rather than hidden: it would not notice a gate that
swapped which of the two methods it reported. Fixing it needs a stable log name
or a glob in the driver, both outside this PR.

All four new manifests were invalid against the v2 schema on first write, and
would have put gate-22 into a permanent FAIL on eight new arms. Fixed in
d3b662f — every new manifest now validates clean. Harmless while every bundle
grades per-gate expectations; the moment a suite adds a run-level-green
assertion it detonates. The hazard is now note 3 in UNCOVERED.md.

Verification

  • bash hydra-gates/tests/run-helper-suites.sh80 discovered, 78 passed, 2
    quarantined, 0 failed
    , with NODE_PATH pointing at an ajv install (the same
    thing CI's npm install --no-save ajv ajv-formats step provides).
  • test_gate_acceptance_matrix.sh — coverage ratchet intact, 58 of 65 gates
    fixtured
    , no gate in both registries, no declared gate in neither.
  • Thirteen sibling suites re-run individually to source every COVERED-ELSEWHERE
    quote.

No checker, no runner, no existing bundle, and no expect.conf outside the five
new directories was modified.

🤖 Generated with Claude Code

@rubenvdlinde
rubenvdlinde merged commit f7dee84 into main Aug 12, 2026
22 of 23 checks passed
@rubenvdlinde
rubenvdlinde deleted the fixtures/uncovered-gate-bundles branch August 12, 2026 21:01
rubenvdlinde added a commit that referenced this pull request Aug 12, 2026
…he subject too (#409)

* test(gate-acceptance): a sibling finding in the SAME file satisfies the subject too

Revert-audits the last two bundles nobody had ever tested. #404 closed asking
for exactly this, on the grounds that #380's remedy -- one mechanism per named
file -- is necessary but not sufficient.

debug-and-conflict is SOUND and is not changed. Four independent reverts, four
predictions written first, four matched: dropping \s* from _CALL_RX reddens the
Debugger.php row alone (120/1); narrowing _CONSTRUCT_RX to (?=[(]) reddens the
Terminator.php row alone (120/1) -- disjoint findings, which is what proves the
two rows separable; exempting all Markdown reddens conflicted.md (120/1); and
reverting the setext discriminator reddens the CLEAN arm instead (119/2), so
both directions are covered.

a11y-noise was NOT. It went green over the exact regression it exists to pin.
FacesCamel.vue was authored to hold "ONLY the forms the \b-anchored noun list
could NOT see", but avatarUrl and avatar_url are TWO code paths, not one:
_CAMEL_1/_CAMEL_2 versus _NON_ALNUM. Reverting camelCase splitting alone left
avatar_url firing, which kept the file in the log, which satisfied grep -qF:
9 findings -> 5 and the suite printed ALL PASS. Two more of the same shape --
dropping plain `src` support (9 -> 8) and single-quoted tabindex (3 -> 2) --
were also green. Every FULL revert was caught, which is the test anybody runs,
and is why this survived three audits.

The invariant needs one more turn. #404's defect was substring containment
BETWEEN subjects; this one is sibling findings INSIDE one subject. The subject
is a file path, and a file path is satisfied by every finding in that file. The
construction that is safe by definition is a named file carrying exactly ONE
finding: any mechanism it depends on empties the file when lost. Where one
finding cannot state the claim, every finding must ride the same code path --
and "same code path" means read the checker, not the fixture's comment.

  a11y-noise  4 -> 7 rows, corpus UNCHANGED (still 9 + 3 findings, redistributed)
    NounsCamel.vue    camelCase only        NounsSnake.vue    avatar_url only
    NounsPlain.vue    the \b-visible nouns  NounsLiteral.vue  the literal src only
    FocusDouble.vue   tabindex="5"          FocusSingle.vue   tabindex='12'

All three formerly-green probes now redden exactly one row each, naming it, so
which row fails tells you which mechanism was lost. render-helpers.js verified
independently: removing js/ts from gate-36's find reddens that row alone.

The planted files were renamed because Faces.vue IS a prefix of FacesCamel.vue
and only the .vue extension separated the two subjects -- relying on an
extension as the terminator is relying on an accident. Pairwise containment is
now checked against the real log text, positive-controlled on #404's own corpus
first, since a grep -qF that cannot match reports "no containment" for free.

Also corrects a false sentence in debug-and-conflict/expect.conf claiming a
regression "fails here on the COUNT". The driver never reads a count. Same
wrong sentence #404 corrected in auth-guards -- found now in two bundles by two
authors, so it is a recurring mental model, not a typo. Fixed in place because
it was load-bearing for somebody's confidence.

And closes the registry class #405 identified: COVERED-ELSEWHERE.md credits
coverage by gate NUMBER while naming the providing suite in PROSE, and nothing
ever resolved that name to a file -- though those rows feed "58 of 65 gates
fixtured". All 19 named suites exist today, so the check pins a true state
rather than reporting a backlog. Built fail-closed: fewer than 10 extracted
names FAILS rather than reporting success over an empty list, because a check
over zero names passes for free, which is the defect it exists to catch.
Proven able to fail by renaming one credited suite -> 130/1.

Thirteen probes, thirteen predictions written first, thirteen matched on count
AND on the identity of every failing assertion. Every revert quoted its actual
changed line from git diff -U0; Edit only, no sed/awk/perl/python. Checkers and
runner restored after every probe and byte-identical to main.

  assertions 121 -> 131 passed / 0 failed

* fix(gate-acceptance): scope the suite-name extraction, and drop a backtick ShellCheck reads as a substitution

Two corrections to the COVERED-ELSEWHERE.md resolution check, both found by CI
rather than by me.

ShellCheck flagged SC2016 on the extraction pattern. The pattern anchored on
the backticks the suite names are written in, and a backtick inside a
single-quoted string reads to ShellCheck as a command substitution. Dropped the
anchor. That is also more correct: some rows carry a scripts/lib/ prefix INSIDE
the backticks, which a backtick-anchored pattern would have missed.

⚠️ The first ShellCheck run had FAILED WITHOUT RUNNING -- the action could not
unpack its download ("does not look like a tar archive") -- so it never saw
this. A peer PR hit the identical transient failure eleven minutes earlier and
went green on re-run. Re-running it here surfaced a REAL finding in my own file
that the infrastructure failure had masked. A red that is the harness and a red
that is the code are worth telling apart in both directions.

Then the broadened pattern counted 21 suites where 20 are credited: the intro
prose names this driver itself. Scoped to table rows, which is where the credit
is actually granted, so the number the check reports means what it says. The
negative control was RE-RUN against the changed extraction rather than inherited
from the earlier version -- renaming a credited suite still reddens exactly one
assertion (130/1).

Merges main, which moved twice under this branch and touched the same driver
(#401's verdict-shape matcher) and the same registry (gates 47 and 48).

  60 of 65 gates fixtured -- 131 passed / 0 failed

---------

Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants