Skip to content

fix(gate-acceptance): a subject that another finding contains is not a subject - #404

Merged
rubenvdlinde merged 1 commit into
mainfrom
fix/acceptance-bundles-revert-proven
Aug 12, 2026
Merged

fix(gate-acceptance): a subject that another finding contains is not a subject#404
rubenvdlinde merged 1 commit into
mainfrom
fix/acceptance-bundles-revert-proven

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

A bundle that stays green over a reverted fix is not evidence of anything

Three of the ten acceptance bundles could not detect the regression they exist to
prevent. All ten go red on a full revert — which is the test anybody actually runs,
and which is why nobody noticed. Three were blind to a partial one, and one of those
was blind to precisely the defect it was built for.

This PR touches only hydra-gates/scripts/test-fixtures/gate-acceptance/. No checker,
no runner, no driver. The driver (test_gate_acceptance_matrix.sh) needed no change at
all — see "the anchor" below.

The general fix: close the subject on the RIGHT

#380's remedy was one mechanism per named file. That is a special case, and it would
not have caught authn-vs-authz, whose three plants legitimately belong in one
controller — the fixture's whole argument is that they differ from each other only by
the preamble.

The real invariant is that a subject must be unsatisfiable by every OTHER finding the
bundle can produce.
_names is grep -qF, and gate-7's log line is
path:LINE method=NAME rule=RULE. So method=preamble is contained in
method=preambleForbiddenCode, and a method name is a prefix of every longer method name.
Every subject here is now closed on the right by its rule=… / calls a service method
field, with one row per arm.

Three controls, run before any edit:

grep -qF 'method=preamble rule=no-auth-guard-in-body'  <preambleForbiddenCode line>  -> rc=1  ✅
grep -qF 'method=preamble rule=no-auth-guard-in-body'  <preamble line>               -> rc=0  ✅
grep -qF 'method=preamble'                             <preambleForbiddenCode line>  -> rc=0  🔴 the bug

⚠️ A trailing space would not have worked — read strips trailing IFS whitespace from
the last field, so method=preamble collapses back to method=preamble and the bundle
stays vacuous while looking fixed. The anchor has to be a token.

What changed

bundle rows mechanisms named files
authn-vs-authz 1 → 3 3 (bare · 401 preamble · 403 preamble) 1
auth-guards 1 → 2 (was the file path) 1 × 2 instances 1
exception-translation 1 → 2 2 (destroy header-binding · purge docblock span) 1
🆕 comment-silenced-guard 2 3 2

🆕 comment-silenced-guard — the third gate-7 blindness had no fixture at all

Of the three ways gate-7 was proven blind, the comment-silenced guard helper was
covered only by unit tests. That is exactly the layer this suite's own header argues is
insufficient:

gate-7 has 86 unit tests. All 86 passed. The gate was still wrong.

Three mechanisms, two named files, three separate reverts:

  • A — the body test reads comment-free text. hermiq's loadOwnedAgent() contains
    no throw statement at all; the word appears only in a comment explaining why it
    catches one, and that sentence cleared all four routed methods.
  • B — a throw must name an authorisation exception. opencatalogi's
    getObjectService()36 controller files across 7 apps — throws when OpenRegister
    is absent, and so cleared every caller.
  • C — an ownership comparison answered with null is recognised, so the narrowing
    costs no true clear. This one only shows in the clean/ arm.

Both arms keep the comment and keep the locator verbatim. #373 did not make prose
illegal, it made prose uncountable — an arm that had also deleted the sentence would
have passed against the broken checker too.

Structurally modelled on gate-65's probe suite: greps a rule name, never a file name,
so a no-op plant leaves the fixture clean and the probe goes red.

Every repair is proven by reverting the fix it defends

Twelve probes. Twelve predictions written before the run. Twelve matched exactly
count and the identity of every failing assertion. Baseline 52 → 70 assertions, 0
failed
.

probe revert predicted got
1 #365, 401 half only — the "just delete 401 from it" repair 69/1, method=preamble
2 #365, 403 half only (the mirror) 69/1, method=preambleForbiddenCode
3 #365 full 68/2, bare survives as liveness control
4 drop multi-parameter methods → lose update() 69/1
5 off-by-one, lose first finding per file 62/5, all five named in advance
6 gate-49 METHOD_RE unbounded docblock 66/2
7 _method_bodies(src)[1:] → lose destroy() 69/1
8 [:1] → lose purge() 69/1
9 mechanism A (body = src[...]) 69/1, method=rotate
9b pure-prose control — A reverted and the comment deleted back to 70/0
10 mechanism B (bare \bthrow\b) 69/1, method=show
11 mechanism C 68/2, clean/ arm, FAIL — 2 method(s)
12 A+B+C 66/4, clean shows 1 method(s)

Probes 4 and 7 are the two the vacuity audit measured at 28 passed / 0 failed —
green over the exact defect.
They are now 1 failure each, naming the lost plant.

Probe 9b matters on its own: gate-7's routed-method scan still reads comment-bearing
text, so an explanatory docblock in a fixture can clear the very method the fixture
plants
. With mechanism A reverted, deleting only the two comment lines restored the
finding — proving the silencer is the comment and nothing else in the file. Every
per-method docblock in the new fixture is austere for that reason; the explanation lives
in the file header, outside every method body span.

⚠️ A trap for the next person adding a bundle

The new fixture's first run reported NOT APPLICABLE — scope was empty — 0 lib/Controller PHP file(s) because the files were untracked — gate-7 enumerates via git ls-files.
An untracked bundle is not a failing bundle, it is an absent one, and its clean/ arm
goes green because nothing was inspected. Stage a new bundle before measuring it.

Scope, honestly

  • publicpage-scope, contract-coverage, license-triangle, prose-not-proof,
    relation-dialect were audited sound and are untouched.
  • a11y-noise and debug-and-conflict (#380) have not been revert-audited by
    anyone.
    They were authored with the one-mechanism-per-file remedy, which this PR shows
    is necessary but not sufficient. Next job.
  • The wider comment leak is still open by designscan_file matches _GUARD_BODY_RE
    against comment-bearing text, so a comment in a routed method containing
    Http::STATUS_FORBIDDEN / , 403) still clears it. This bundle pins the helper path
    only, and says so in its header.
  • Coverage is unchanged at 22 of 65 gates fixtured. "The acceptance matrix is green"
    is still a statement about 22 gates, not 65.

🤖 Generated with Claude Code

…a subject

Three of the ten acceptance bundles could not detect the regression they exist
to prevent. All ten go red on a FULL revert -- which is the test anybody
actually runs, and which is why nobody noticed. Three were blind to a PARTIAL
one, and one of those was blind to precisely the defect it was built for.

`_names` is `grep -qF`. gate-7's log line is `path:LINE method=NAME rule=RULE`,
so `method=preamble` is CONTAINED IN `method=preambleForbiddenCode`. Reverting
only the 401 half of #365 removed preamble() from the log entirely and the suite
printed `PASS -- gate-7 NAMES 'method=preamble'` and `ALL PASS`.

#380's remedy -- one mechanism per named file -- would not have caught that:
all three plants legitimately belong in one controller, because the fixture's
whole argument is that the arms differ only by the preamble. The real invariant
is that a subject must be unsatisfiable by every OTHER finding the bundle can
produce. A method name is a prefix of every longer method name, so a name alone
is never a discriminator. Every subject is now closed on the right by its
` rule=...` / ` calls a service method` field, one row per arm.

A trailing space would NOT have worked: `read` strips trailing IFS whitespace
from the last field, so `method=preamble ` collapses back and the bundle stays
vacuous while looking fixed. The anchor has to be a token. Internal spaces DO
survive, which is why the driver needed no change.

  authn-vs-authz        1 -> 3 rows  (bare / 401 preamble / 403 preamble)
  auth-guards           1 -> 2 rows  (was the FILE PATH, which both plants name)
  exception-translation 1 -> 2 rows  (destroy header-binding / purge docblock)
  comment-silenced-guard    NEW      3 mechanisms, 2 files, 2 rows

The new bundle closes the third of gate-7's three proven blindnesses, which had
no repo-shaped fixture at all -- only unit tests, i.e. exactly the layer this
suite's own header argues is insufficient ("gate-7 has 86 unit tests. All 86
passed. The gate was still wrong."). Its three mechanisms revert separately:

  A  the body test reads comment-free text.  hermiq's loadOwnedAgent() contains
     no `throw` statement at all -- the word appears only in a comment
     explaining why it CATCHES one -- and that cleared all four routed methods.
  B  a `throw` must NAME an authorisation exception.  opencatalogi's
     getObjectService(), in 36 controller files across 7 apps, throws when
     OpenRegister is absent and so cleared every caller.
  C  an ownership comparison answered with `null` IS recognised, so the
     narrowing costs no true clear.  Visible only in the clean/ arm.

Both arms keep the comment and keep the locator verbatim. #373 did not make
prose illegal, it made prose uncountable; an arm that also deleted the sentence
would have passed against the broken checker too.

Twelve probes, twelve predictions written before the run, twelve matched exactly
-- count and the identity of every failing assertion. 52 -> 70 assertions, 0
failed. Probes 4 and 7 are the two the vacuity audit measured at 28 passed /
0 failed, green over the exact defect; they are now one failure each, naming
the lost plant.

Probe 9b is its own result: gate-7's routed-method scan still reads
comment-bearing text, so an explanatory docblock in a fixture can clear the very
method the fixture plants. With mechanism A reverted, deleting ONLY the two
comment lines restored the finding -- so the silencer is the comment and nothing
else in the file. Every per-method docblock in the new fixture is austere for
that reason.

Not closed, and said so in the bundle header: `scan_file` still matches
_GUARD_BODY_RE against comment-bearing text, so a comment in a ROUTED method
containing Http::STATUS_FORBIDDEN / `, 403)` still clears it. This bundle pins
the helper path only. a11y-noise and debug-and-conflict (#380) have not been
revert-audited by anyone. Coverage is unchanged at 22 of 65 gates fixtured.

Only hydra-gates/scripts/test-fixtures/gate-acceptance/ is touched. No checker,
no runner, no driver.
@rubenvdlinde
rubenvdlinde merged commit 8167939 into main Aug 12, 2026
33 checks passed
@rubenvdlinde
rubenvdlinde deleted the fix/acceptance-bundles-revert-proven branch August 12, 2026 21:11
rubenvdlinde added a commit that referenced this pull request Aug 12, 2026
…he subject too (#409)

* test(gate-acceptance): a sibling finding in the SAME file satisfies the subject too

Revert-audits the last two bundles nobody had ever tested. #404 closed asking
for exactly this, on the grounds that #380's remedy -- one mechanism per named
file -- is necessary but not sufficient.

debug-and-conflict is SOUND and is not changed. Four independent reverts, four
predictions written first, four matched: dropping \s* from _CALL_RX reddens the
Debugger.php row alone (120/1); narrowing _CONSTRUCT_RX to (?=[(]) reddens the
Terminator.php row alone (120/1) -- disjoint findings, which is what proves the
two rows separable; exempting all Markdown reddens conflicted.md (120/1); and
reverting the setext discriminator reddens the CLEAN arm instead (119/2), so
both directions are covered.

a11y-noise was NOT. It went green over the exact regression it exists to pin.
FacesCamel.vue was authored to hold "ONLY the forms the \b-anchored noun list
could NOT see", but avatarUrl and avatar_url are TWO code paths, not one:
_CAMEL_1/_CAMEL_2 versus _NON_ALNUM. Reverting camelCase splitting alone left
avatar_url firing, which kept the file in the log, which satisfied grep -qF:
9 findings -> 5 and the suite printed ALL PASS. Two more of the same shape --
dropping plain `src` support (9 -> 8) and single-quoted tabindex (3 -> 2) --
were also green. Every FULL revert was caught, which is the test anybody runs,
and is why this survived three audits.

The invariant needs one more turn. #404's defect was substring containment
BETWEEN subjects; this one is sibling findings INSIDE one subject. The subject
is a file path, and a file path is satisfied by every finding in that file. The
construction that is safe by definition is a named file carrying exactly ONE
finding: any mechanism it depends on empties the file when lost. Where one
finding cannot state the claim, every finding must ride the same code path --
and "same code path" means read the checker, not the fixture's comment.

  a11y-noise  4 -> 7 rows, corpus UNCHANGED (still 9 + 3 findings, redistributed)
    NounsCamel.vue    camelCase only        NounsSnake.vue    avatar_url only
    NounsPlain.vue    the \b-visible nouns  NounsLiteral.vue  the literal src only
    FocusDouble.vue   tabindex="5"          FocusSingle.vue   tabindex='12'

All three formerly-green probes now redden exactly one row each, naming it, so
which row fails tells you which mechanism was lost. render-helpers.js verified
independently: removing js/ts from gate-36's find reddens that row alone.

The planted files were renamed because Faces.vue IS a prefix of FacesCamel.vue
and only the .vue extension separated the two subjects -- relying on an
extension as the terminator is relying on an accident. Pairwise containment is
now checked against the real log text, positive-controlled on #404's own corpus
first, since a grep -qF that cannot match reports "no containment" for free.

Also corrects a false sentence in debug-and-conflict/expect.conf claiming a
regression "fails here on the COUNT". The driver never reads a count. Same
wrong sentence #404 corrected in auth-guards -- found now in two bundles by two
authors, so it is a recurring mental model, not a typo. Fixed in place because
it was load-bearing for somebody's confidence.

And closes the registry class #405 identified: COVERED-ELSEWHERE.md credits
coverage by gate NUMBER while naming the providing suite in PROSE, and nothing
ever resolved that name to a file -- though those rows feed "58 of 65 gates
fixtured". All 19 named suites exist today, so the check pins a true state
rather than reporting a backlog. Built fail-closed: fewer than 10 extracted
names FAILS rather than reporting success over an empty list, because a check
over zero names passes for free, which is the defect it exists to catch.
Proven able to fail by renaming one credited suite -> 130/1.

Thirteen probes, thirteen predictions written first, thirteen matched on count
AND on the identity of every failing assertion. Every revert quoted its actual
changed line from git diff -U0; Edit only, no sed/awk/perl/python. Checkers and
runner restored after every probe and byte-identical to main.

  assertions 121 -> 131 passed / 0 failed

* fix(gate-acceptance): scope the suite-name extraction, and drop a backtick ShellCheck reads as a substitution

Two corrections to the COVERED-ELSEWHERE.md resolution check, both found by CI
rather than by me.

ShellCheck flagged SC2016 on the extraction pattern. The pattern anchored on
the backticks the suite names are written in, and a backtick inside a
single-quoted string reads to ShellCheck as a command substitution. Dropped the
anchor. That is also more correct: some rows carry a scripts/lib/ prefix INSIDE
the backticks, which a backtick-anchored pattern would have missed.

⚠️ The first ShellCheck run had FAILED WITHOUT RUNNING -- the action could not
unpack its download ("does not look like a tar archive") -- so it never saw
this. A peer PR hit the identical transient failure eleven minutes earlier and
went green on re-run. Re-running it here surfaced a REAL finding in my own file
that the infrastructure failure had masked. A red that is the harness and a red
that is the code are worth telling apart in both directions.

Then the broadened pattern counted 21 suites where 20 are credited: the intro
prose names this driver itself. Scoped to table rows, which is where the credit
is actually granted, so the number the check reports means what it says. The
negative control was RE-RUN against the changed extraction rather than inherited
from the earlier version -- renaming a credited suite still reddens exactly one
assertion (130/1).

Merges main, which moved twice under this branch and touched the same driver
(#401's verdict-shape matcher) and the same registry (gates 47 and 48).

  60 of 65 gates fixtured -- 131 passed / 0 failed

---------

Co-authored-by: Conduction Release Bot <release-bot@conduction.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant