fix(gate-16): an EMPTY SCOPE is not a PASS (.github#361) - #364
Merged
Conversation
gate-16 is diff-scoped BY DESIGN (ADR-020). On a full-repo run there is no diff to scope to: the runner passed BASE_REF unguarded, the checker fell back to its `origin/development` default, and on `development` itself that diffs the branch against itself -- `# count=0` -> PASS over a scope nothing was ever read from. Worse than the sibling #347: gate-61's NOT APPLICABLE is excluded from the verdict, but gate-16's PASS COUNTS toward 'N of N applicable gates ran'. It does not hide a number, it falsifies a verdict. Caught live on openregister#2422, where gates 19/25/26 printed NOT APPLICABLE and gate-16 printed PASS four lines apart over the same empty scope. Measured in five repos, one tree each, changing only this input: pipelinq 0 / 185 (beta) / 1466 (report) openregister 0 / 232 (beta) / 234 docudesk 0 / 290 (beta) / 471 shillinq 0 / 808 (empty-tree base) procest 0 / 113 (beta) / 445 larpingapp 0 / 0 / 0 <- genuinely clean, positive-controlled gate-19 already guards this (the #242 fix, ~285 lines below); gate-16 was never given the same treatment. DELIBERATELY NOT a literal copy of gate-19's else-branch. gate-19 falls back to a FULL SWEEP; for gate-16 that would flag the entire legacy @SPEC surface -- the wrong contract per ADR-020, and a false RED in every repo. Reporting NOT APPLICABLE with a reason naming the ABSENCE of a diff is honest and cannot go false-RED. VERIFICATION full run, before: [gate-16] spec-coverage: PASS full run, after: [gate-16] spec-coverage: NOT APPLICABLE - full-repo run computed NO diff ... This is NOT a pass. checker positive control on a real tree (openregister): HYDRA_GATE_BASE_REF=origin/beta -> # count=232 HYDRA_GATE_BASE_REF=origin/development -> # count=0 The diff-scoped arm is BYTE-IDENTICAL to the original invocation -- the diff shows the same two lines moved inside an `if`, nothing else -- so behaviour under --scope-to-diff is unchanged by construction. `_skip` category is `na`: anything outside na|structural|wiring is treated as an internal-error FAIL. My first draft passed `scope` and would itself have produced the false RED this change exists to prevent.
…ope as a verdict The package's own test_gate_crashed_checker_is_not_a_finding.sh caught a regression in my first draft, which is exactly what it exists for. That draft skipped the invocation entirely on a full run. A CRASHED checker then became invisible at full scope -- nothing ran, so nothing could crash -- and gate-16 reported NOT APPLICABLE where the invariant requires SKIPPED (wiring) naming what went unverified: FAIL — gate-16 spec-coverage: verdict is 'NOT' — expected SKIPPED (wiring) FAIL — gate-16's skip does not say what went unchecked Now the checker ALWAYS runs, on every scope, and the two conditions are evaluated in the order that keeps them distinguishable: 1. WIRING — did it finish? (no terminal '# count=' marker -> SKIPPED) 2. SCOPE — is its answer meaningful? (full run -> NOT APPLICABLE) A crash must never be reported as an empty scope, and an empty scope must never be reported as a pass. Re-run after the change: ok — gate-16 spec-coverage: SKIPPED (wiring) when python3 cannot run ok — gate-16's skip names what it left unverified ok — control: gate-16 produces a real verdict with a working python3
rubenvdlinde
pushed a commit
that referenced
this pull request
Aug 11, 2026
#364 changed a verdict with no fixture behind it. This adds one, driven through the real bin/hydra-gates over a purpose-built two-commit repository, and observed RED before the change and GREEN after: before (81c8c97) 3 failed / 9 passed -- arm 3: "#361 is LIVE: gate-16 printed PASS on a --full run that computed no diff, over a tree whose two untagged methods the positive control just named" after (274b926) 0 failed / 12 passed Both of #364's load-bearing claims were re-derived here rather than taken on trust, by patching the runner and re-running this same fixture: * category `scope` instead of `na` -> [gate-16] FAIL - internal error: _skip called with reason category 'scope'. A fleet-wide false RED, exactly as the PR body says. The suite now asserts against it by name. * a full sweep on --full -> [gate-16] FAIL - 2 changed method(s) missing @SPEC, naming LegacyDebtController, which no author touched. ADR-020 violated. The suite also asserts the legacy surface never reaches the log, so a sweep cannot creep back in wearing a NOT APPLICABLE. REFINEMENT to the second claim, for the record: a LITERAL copy of gate-19's else-branch (dropping HYDRA_GATE_BASE_REF=) would not have swept at all -- check_spec_coverage.py:508 defaults the env lookup to 'origin/development', so it returns `# count=0` and changes nothing. The false-RED danger is real but arrives one line later, with the check_visual_coverage.py `or None` precedent. The arms, all through bin/hydra-gates on one tree: positive control --mode report names LegacyDebtController, 2 uncovered arm 1 diff, docs only PASS (ADR-020: inherited debt is not mine) arm 2 diff ADDS untagged work FAIL, and NAMES NewWorkController.php arm 3 --full, arm 1's tree NOT APPLICABLE, reason names the ABSENCE of a diff, no legacy finding in the log, gate-16 listed in the coverage summary's NA block Coverage ratchet: gate-16 moves from UNCOVERED.md (`needs-diff`) to COVERED-ELSEWHERE.md. 12 of 64 fixtured -> 13 of 64.
rubenvdlinde
added a commit
that referenced
this pull request
Aug 11, 2026
#364 changed a gate verdict fleet-wide and shipped with NO fixture behind it. That violated the rule this programme applies to every other gate change, and the omission was mine as the author. This is the fixture it should have carried. Driven through the real bin/hydra-gates over a purpose-built two-commit repository, and OBSERVED RED BEFORE the change and GREEN AFTER: 81c8c97 (pre-#364) 3 failed / 9 passed arm 3: '.github#361 is LIVE: gate-16 printed PASS on a --full run that computed no diff, over a tree whose two untagged methods the positive control just named' 765d081 (post-#364) 0 failed / 12 passed Both of #364's load-bearing claims were RE-DERIVED rather than taken on trust, each by patching the runner and re-running this same fixture: - '_skip 16 ... scope' instead of 'na' produces '[gate-16] spec-coverage: FAIL - internal error: _skip called with reason category scope' -- confirming the first draft would have been a fleet-wide false RED, not merely wrong. - copying gate-19's else-branch (a full sweep) instead of NOT APPLICABLE confirms the second claim: for gate-16 that flags the entire legacy @SPEC surface, the wrong contract per ADR-020. A fix asserting 'this cannot go false-RED' is exactly the kind of claim that needs its own control, and #364 asserted it without one.
Contributor
Author
A seventh repo for the table, and it looks like larpingapp, not like pipelinqMeasured after this merged, but with the pre-fix package (
Positive-controlled, committed — an untagged public method in Two things worth adding to the record:
On "why not a literal copy of gate-19's guard"Agreed, and the asymmetry is worth naming explicitly for whoever reads this So the distinguishing property was never "does the checker default the env |
This was referenced Aug 11, 2026
rubenvdlinde
added a commit
that referenced
this pull request
Aug 12, 2026
…ented a count Two repairs, both from the same rule: a crashed checker must never read as an empty scope, and an empty scope must never read as a pass. gate-61 — the first draft of the delta-base change skipped the checker invocation entirely when no base was resolved and short-circuited to the empty-scope status. Nothing ran, so nothing could crash, and test_gate_crashed_checker_is_not_a_finding.sh caught it immediately: with a python3 that cannot run, gate-61 reported NOT APPLICABLE where it must report SKIPPED (wiring). That is the same regression #364's first draft made in gate-16, caught by the same suite, and the repair is the same one: always run the checker, evaluate wiring first and scope second. With no base the helper is now invoked in its whole-tree mode purely to learn whether it can run, and its findings are discarded — sweeping on a baseless run was tried before and reverted because the builder runs that way. gate-24 — the parity wrapper is app-owned and declares no terminal marker, so a wrapper that exited non-zero having written nothing parseable was reported as one parity violation: a fabricated number with a plausible message and nothing behind it, the same shape as gates 22, 25 and 28. The verdict deliberately stays FAIL, because nothing here can tell a crash from a differently-phrased finding and guessing wiring would turn a real parity failure into a green hole. It simply stops claiming a count nobody measured. gate-61 carries the same clamp idiom and it is left alone with the reason written down: its wiring branch sits immediately above, so a crash cannot reach it, and zero counted lines with a non-zero exit is format drift rather than a dead checker. Two suites updated, neither weakened. The crashed-checker suite now accepts gate-17's NOT APPLICABLE on its E2BIG arm only when the terminal count marker proves the interpreter ran over the oversized scope list, which is a stronger assertion than the two it sits beside; its verdict regex also gained the multi-word form that made NOT APPLICABLE parse as NOT. The route-auth suite's gate-14 expectation moves from PASS to NOT APPLICABLE for the same reason its gate-5 neighbour moved in August: a pass here is scoping, not absence, and that reclassification was only half done. Refs: .github#374, #330, #364, #379
rubenvdlinde
added a commit
that referenced
this pull request
Aug 12, 2026
* feat(gates)!: scan the whole tree by default — ADR-020 superseded The gates now scan the ENTIRE tracked codebase. Diff scoping becomes opt-in via --scope-to-diff / --diff / HYDRA_GATE_SCOPE=diff. Ruben's rationale: when a gate changes or is added, the next push to beta should fail unless the old code is brought up to the new standard, so gate updates travel with releases instead of applying only to code written after them. Recorded in hydra-gates/ADR-020-SUPERSEDED.md, including what it costs: the first development-to-beta run after this lands surfaces the fleet backlog at once, and the last wide-scope measurement was roughly 3,900 findings. The scope is now two independent, named inputs rather than one. File scope controls which files the state gates open and defaults to full; the delta base is resolved separately and is what the five delta gates compare against. Gates 16, 29, 47, 48 and 61 are keyed on whether a base was resolved, not on the file scope, so a pull request keeps every delta gate and gains whole-tree state coverage. Without a base they report NOT APPLICABLE by name, never PASS. An unresolvable base stays fatal on a diff-scoped run and is non-fatal at full scope, where it costs five gates rather than all sixty-four. BASE_REF no longer carries a hardcoded origin/development default, which was the shared root cause of the gate-16 and gate-61 empty-scope defects. Eighteen gates could report PASS over a scope they never opened. Full scope removes the empty set in the common case but leaves the fall-through latent, so it is fixed at the fall-through through one shared helper: gates 14, 17, 18, 20, 21, 22, 34 through 44 and 52. Gates 14 and 20 were found by sweeping the table and are not in the issue's list of sixteen. Gate 40's inline note arguing against this was false on both of its claims and is deleted with the fix; gate 52's uncomputed ratchet half and gate 53's uncomputed orphan-promotion half now say so on stdout instead of hiding behind a PASS. APP_DIR is absolutised before the cd and the resolved absolute path is printed. Gate 17 was the only gate handing APP_DIR to its checker afterwards, so a relative app-dir made the scan root resolve a second time against the app dir itself: same tree, absolute path FAIL 1, relative path PASS. CI was never exposed because bin/hydra-gates absolutises; the documented human invocation was. The acceptance suite now enforces the empty-scope property across the package instead of for seven named gates, and carries a relative-path arm the standing driver is structurally unable to express because it always builds absolute paths. Both new arms were proven able to fail by deliberate breakage. Helper suites 74 passed / 2 quarantined / 1 failed before and after, the failure being the pre-existing unresolvable-ajv needs-external skip that fails closed locally and is green in CI. Entry-point tests 61 passed / 0 failed before, 69 passed / 0 failed after. Refs: .github#374, #361, #347, #364, #371 * docs(gates): the runner header described the old default The Options block still said --scope-to-diff was the mode and named origin/development as the base default. Both are gone: full scope is the default, and BASE_REF starts empty because that hardcoded default is the shared root cause of .github#347 and #361. * fix(gates): a crashed gate-61 read as an empty scope, and gate-24 invented a count Two repairs, both from the same rule: a crashed checker must never read as an empty scope, and an empty scope must never read as a pass. gate-61 — the first draft of the delta-base change skipped the checker invocation entirely when no base was resolved and short-circuited to the empty-scope status. Nothing ran, so nothing could crash, and test_gate_crashed_checker_is_not_a_finding.sh caught it immediately: with a python3 that cannot run, gate-61 reported NOT APPLICABLE where it must report SKIPPED (wiring). That is the same regression #364's first draft made in gate-16, caught by the same suite, and the repair is the same one: always run the checker, evaluate wiring first and scope second. With no base the helper is now invoked in its whole-tree mode purely to learn whether it can run, and its findings are discarded — sweeping on a baseless run was tried before and reverted because the builder runs that way. gate-24 — the parity wrapper is app-owned and declares no terminal marker, so a wrapper that exited non-zero having written nothing parseable was reported as one parity violation: a fabricated number with a plausible message and nothing behind it, the same shape as gates 22, 25 and 28. The verdict deliberately stays FAIL, because nothing here can tell a crash from a differently-phrased finding and guessing wiring would turn a real parity failure into a green hole. It simply stops claiming a count nobody measured. gate-61 carries the same clamp idiom and it is left alone with the reason written down: its wiring branch sits immediately above, so a crash cannot reach it, and zero counted lines with a non-zero exit is format drift rather than a dead checker. Two suites updated, neither weakened. The crashed-checker suite now accepts gate-17's NOT APPLICABLE on its E2BIG arm only when the terminal count marker proves the interpreter ran over the oversized scope list, which is a stronger assertion than the two it sits beside; its verdict regex also gained the multi-word form that made NOT APPLICABLE parse as NOT. The route-auth suite's gate-14 expectation moves from PASS to NOT APPLICABLE for the same reason its gate-5 neighbour moved in August: a pass here is scoping, not absence, and that reclassification was only half done. Refs: .github#374, #330, #364, #379 * fix(gate-61): the scope-matrix invariant read the FILE scope, not whether a diff existed The gate-agnostic property — a NOT APPLICABLE may not blame a diff on a run that computed none — is exactly right and stays. Its TRIGGER was wrong: it keyed on --full, when the property is about whether a diff was COMPUTED. Those are two independent inputs now, so the suite was reading the wrong one. That is the same conflation .github#347 was, one layer up, inside the test written to catch it. It fired on this branch and it fired correctly. Arm 3 now removes the BASE rather than the file scope, so every assertion it already carried holds verbatim. Arm 3b covers what used to be unreachable and is now what every PR in the fleet runs: full file scope with a real base. There, naming the diff is a true statement, and what has to hold instead is the falsifiability condition whose absence made #347 unreadable for weeks — a reason may name a diff only on a run that PRINTED the base it diffed against, so a reader can check the claim rather than trust it. Asserted from both sides. The advisory whole-tree sweep now runs in BOTH empty-scope branches. It ran only on the no-base path, because that used to be the only way to reach a state where the size of the unread backlog mattered. Full scope is the default now, so "a base resolved and excluded every registration" is the common case, and the ambiguity the sweep was added to remove had simply moved to the path everyone runs: 0 of 1 and 0 of 45 still printed identically there. The whole point of the reversal is that inherited debt stops being invisible, and leaving this gate's backlog unstated on the common path would have exempted it from exactly that. Refs: .github#374, #347 * ci(gates): the unresolvable-base check asserted the contract this PR supersedes The required "Unresolvable base ref must exit 99 with no green" step encoded ADR-020's rule as the ONLY acceptable behaviour, so this PR could not merge: under the new default an unresolvable base no longer aborts, it falls back to full scope. The old assertion is kept BYTE-FOR-BYTE and simply run in the mode it was always about (--scope-to-diff): with an explicit diff request and no usable base, the runner must still exit 99, print NOTHING WAS CHECKED, and emit no gate lines. Every clause of it still holds. A second step covers the new default: the run must say the base was unresolvable, must still emit gate lines, and none of gates 16/29/47/48/61 may print PASS over a scope that was never read. Positive-controlled against this runner's real captured output four ways — injecting "[gate-16] ... PASS" fails it, deleting a gate line fails it, and the diff-mode arm fails if the 99/NOTHING WAS CHECKED behaviour regresses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #361.
gate-16 is diff-scoped by design (ADR-020). On a full-repo run there is no diff, so the runner's unguarded
BASE_REFlet the checker fall back toorigin/development— which ondevelopmentdiffs the branch against itself. Result:# count=0→ PASS over a scope nothing was read from.This is worse than the sibling #347. gate-61's
NOT APPLICABLEis excluded from the verdict; gate-16'sPASScounts toward "N of N applicable gates ran". It does not hide a number — it falsifies a verdict. Caught live on openregister#2422, where gates 19/25/26 printedNOT APPLICABLEand gate-16 printedPASSfour lines apart over the same empty scope.Measured in six repos, one tree each, changing only this input
larpingapp matters: it proves this is not a blanket "every repo hides debt" claim.
Why not a literal copy of gate-19's guard
gate-19's else-branch falls back to a full sweep. For gate-16 that would flag the entire legacy
@specsurface — the wrong contract per ADR-020, and a false RED in every repo. ReportingNOT APPLICABLEwith a reason naming the absence of a diff is honest and cannot go false-RED.Verification
[gate-16] spec-coverage: PASS→ after:NOT APPLICABLE — full-repo run computed NO diff … This is NOT a pass.origin/beta→# count=232,origin/development→# count=0if), so--scope-to-diffbehaviour is unchanged by constructionscopeas the_skipcategory. Anything outsidena|structural|wiringis treated as an internal-error FAIL — it would have produced exactly the false RED this change prevents. Nowna.Lands after #359 (the acceptance suite), which is merged.