fix(gates): nine gates that turned a crashed helper into a verdict — 6, 7, 9 and 59-64 by rubenvdlinde · Pull Request #330 · ConductionNL/.github

rubenvdlinde · 2026-08-10T10:29:46Z

Closes the last live instances of the construct that made gate-20 blind for its entire existence — a trailing 2>/dev/null || true on a helper invocation, which discards the helper's message and its exit status — and the same defect in its red-shifted form, where a crashed helper was rendered as a finding count nobody had counted.

Band 1 — the construct itself (gates 6, 7, 9)

Three gates invoked their Python helper as >> log 2>/dev/null || true and then derived the verdict from wc -l on the log. A helper that never started wrote nothing, the count was 0, and the gate printed PASS.

Measured at package 71e01bf, one fixture, two runs:

gate	working `python3`	`python3` that exits 1
gate-6 `orphan-auth`	FAIL — 1 orphan method	PASS
gate-7 `no-admin-idor`	FAIL — 1 unguarded method	PASS
gate-9 `semantic-auth`	FAIL — 1 mismatch	PASS

Same tree, same files in scope. Three security gates — dead authorization code, IDOR (OWASP A01:2021), auth-attribute-vs-body mismatch — certifying a tree they had not opened. gate-7 is the gate a missing helper had already blinded once over 11 real unguarded endpoints (#147); 2>/dev/null || true left the second route to the identical false green wide open.

All three helpers end return 0 # exit 0 always — caller counts printed lines, so a non-zero byte here can only be a crash and never a finding count (#209). stderr now lands in a .err sibling and a non-zero status produces a named _skip <n> ... wiring quoting it:

[gate-7] no-admin-idor: SKIPPED (wiring) — check_no_admin_idor.py exited 1 — 1 controller
file(s) were in scope and NONE were judged; unguarded #[NoAdminRequired] endpoints (IDOR,
OWASP A01:2021) are UNVERIFIED by this run. This helper always exits 0 when it runs, so this
is a crash, not a finding count. Checker output: Traceback (most recent call last): ...

Band 2 — the mirror image (gates 59, 60, 61, 62, 63, 64)

These six take a numeric exit protocol from their helper (0 pass / 1 findings / 3 empty scope / 4 n-a / 5 tooling missing) and sent every unrecognised code to _fail. A crash also exits 1, so it arrived at the findings branch — and because a traceback carries no FAIL lines, _count '^FAIL' returned 0 and the next line read [ "${_n}" -eq 0 ] && _n=1, inventing a finding count of one for a check that had measured nothing.

gate	working `python3`	`python3` that exits 1
gate-59 `unclosable-gate`	PASS	FAIL — config gate(s) read but never written
gate-62 `store-plane`	NOT APPLICABLE	FAIL — 1 naming or discovery violation(s)
gate-63 `settings-surface`	NOT APPLICABLE	FAIL — 1 naming or discovery violation(s)

62 and 63 blocked a repository that does not have the subject matter at all. A gate wrong in the red direction burns the same credit as one wrong in the green direction — and it is how a suite gets switched off.

The exit byte cannot separate the two, because 1 means both "I found things" and "I died". What can is that every one of these helpers prints a terminal summary line on each path that reaches a verdict — checked N manifest(s): M failure(s), N unclosable gate(s)., apphost-autoload-prelude: OK — and a process that died before its own summary never printed it. That is the evidence gates 15/16/17 already take from their # count= marker (#271), read from summaries these helpers already emit rather than from a marker they would have to grow. _helper_finished states the rule once.

Gates 59 and 64 print no FAIL-prefixed lines at all, so _count '^FAIL' was always 0 and the fallback _n=1 fired on every real failure — a pre-existing invented number. It now comes off the helper's own summary line.

What is deliberately left tolerant

|| true is not deleted wholesale. This runner has errexit off by design (see its header) precisely because a gate returning non-zero is normal, and #243 showed that removing tolerance can abort a run and leave 38 gates unexecuted — a dead run is worse than a blind gate. Every || true kept is either infrastructure whose failure cannot manufacture a verdict (mktemp, mkdir, the provenance banner, grep -c over an empty stream), or a filter whose failure direction is over-reporting (_filter_preexisting: a crash leaves the findings log untouched, never truncated).

Gates 25 and 26 are also left alone: a crashed helper there already yields FAIL — an unreported number of ..., which is non-green and says outright that it did not measure. Not silent, so not in scope for this change.

Proof, both directions

Each fix exercised with the helper alive and with it exiting 1.
Live verdicts unchanged: the full 64-gate verdict list over the same fixture is byte-identical to the pre-fix baseline, so nothing became a skip that was not already one.
--require-full-coverage stays satisfiable: on a fixture with planted true positives the run reports COVERAGE: 34 of 64 declared gates reported a result (30 not applicable; 34 of 34 applicable gates ran) — zero skips, no new coverage trip.
Anti-widening: with a working interpreter, gates 59/60/61/64 still FAIL their planted true positives.
Regression suites green: test_gate_crashed_checker_is_not_a_finding.sh, test_gate_5661_empty_scope_is_not_a_pass.sh, test_gate_empty_scope_never_passes.sh, test_gates_23_33_never_green_over_nothing.sh (39/39), test_gate_errexit_discipline.sh.

Regression coverage

test_gate_crashed_checker_is_not_a_finding.sh gains:

gates 6/7/9 in the generic dead-interpreter arm;
a two-arm arm for 59–64 (planted TPs must still FAIL alive; must be SKIPPED (wiring) dead);
an assertion that gate-59's failure carries a measured count;
a detector for the construct itself, so the next gate landing with 2>/dev/null || true fails on its first run. Positive control: it fires on exactly the three pre-fix sites (2055, 2133, 2266) and is clean afterwards.

Not in this PR

#327 is NOT subsumed. Gates 16 and 20 emitting no verdict line at all on a --full run is a different mechanism — a missing else on an if [ -d lib ] prerequisite, not a crashed helper. Every path this PR touches already emitted a line; it changes which line. #327 remains open and unaffected.

quality.yml is untouched.

🤖 Generated with Claude Code

…6, 7, 9 and 59-64 The gate-20 construct, and its mirror image. BAND 1 — `2>/dev/null || true` (gates 6, 7, 9) Three gates invoked their Python helper as `>> log 2>/dev/null || true` and then took the verdict from `wc -l` on the log. That discards the helper's message AND its exit status, so a helper that never started wrote nothing, the count was 0, and the gate printed PASS. It is the exact mechanism behind gate-20 having NEVER fired in any repo in its entire existence. Measured at package 71e01bf, one fixture, two runs: working python3 python3 that exits 1 [gate-6] orphan-auth: FAIL [gate-6] orphan-auth: PASS [gate-7] no-admin-idor: FAIL [gate-7] no-admin-idor: PASS [gate-9] semantic-auth: FAIL [gate-9] semantic-auth: PASS Three SECURITY gates — dead authorization code, IDOR, auth-attribute-vs-body mismatch — reporting a tree clean that they had not opened. gate-7 is the gate a MISSING helper had already blinded once over 11 real unguarded endpoints (#147); `2>/dev/null || true` left the second route to the same false green. All three helpers end `return 0 # exit 0 always — caller counts printed lines`, so a non-zero byte can only be a crash and never a finding count. stderr now goes to a `.err` sibling and a non-zero status produces a named `_skip <n> ... wiring` quoting the first lines of it. BAND 2 — a crash wearing an invented finding count (gates 59-64) The same disease in red. These six take a numeric exit protocol from their helper (0 pass / 1 findings / 3 empty scope / 4 n-a / 5 tooling missing) and sent every unrecognised code to `_fail`. A crash also exits 1, so it landed in the findings branch — and because a traceback carries no `FAIL` lines, `_count '^FAIL'` returned 0 and the next line read `[ "${_n}" -eq 0 ] && _n=1`, INVENTING a count of one for a check that had measured nothing. working python3 python3 that exits 1 [gate-59] unclosable-gate: PASS FAIL — config gate(s) read but never written [gate-62] store-plane: NOT APPLICABLE FAIL — 1 naming or discovery violation(s) [gate-63] settings-surface: NOT APPL. FAIL — 1 naming or discovery violation(s) 62 and 63 blocked a repo that does not have the subject matter at all. The exit byte cannot separate crash from finding, because 1 means both. What can is that every one of these helpers prints a TERMINAL SUMMARY LINE on each path that reaches a verdict, and a process that died before its own summary never printed it — the evidence gates 15/16/17 already take from `# count=` (#271), read from summaries these helpers already emit. `_helper_finished` holds the rule once. Gates 59 and 64 print no `FAIL`-prefixed lines at all, so their count was ALWAYS the invented 1. It now comes off the helper's own summary line. WHAT IS DELIBERATELY LEFT TOLERANT `|| true` is not deleted wholesale: this runner has no errexit (see the header) precisely so a gate returning non-zero is normal, and #243 showed that removing tolerance can abort a run and leave 38 gates unexecuted. Every `|| true` kept here is on infrastructure whose failure cannot manufacture a verdict — mktemp, mkdir, the provenance banner, `grep -c` on an empty stream — or on a filter whose failure direction is over-reporting, never under. PROOF, BOTH DIRECTIONS Each fix is exercised with the helper alive and with it exiting 1. The live verdicts over the same fixture are byte-identical to the pre-fix baseline, so nothing became a skip that was not one. A --require-full-coverage run on a fixture with planted true positives reports "34 of 34 applicable gates ran" — no new skip trips it. test_gate_crashed_checker_is_not_a_finding.sh gains gates 6/7/9 in its generic dead-interpreter arm, a two-arm arm for 59-64, an assertion that gate-59's count is measured, and a detector for the construct itself — which fires on exactly the three pre-fix sites and is clean afterwards.

…ented a count Two repairs, both from the same rule: a crashed checker must never read as an empty scope, and an empty scope must never read as a pass. gate-61 — the first draft of the delta-base change skipped the checker invocation entirely when no base was resolved and short-circuited to the empty-scope status. Nothing ran, so nothing could crash, and test_gate_crashed_checker_is_not_a_finding.sh caught it immediately: with a python3 that cannot run, gate-61 reported NOT APPLICABLE where it must report SKIPPED (wiring). That is the same regression #364's first draft made in gate-16, caught by the same suite, and the repair is the same one: always run the checker, evaluate wiring first and scope second. With no base the helper is now invoked in its whole-tree mode purely to learn whether it can run, and its findings are discarded — sweeping on a baseless run was tried before and reverted because the builder runs that way. gate-24 — the parity wrapper is app-owned and declares no terminal marker, so a wrapper that exited non-zero having written nothing parseable was reported as one parity violation: a fabricated number with a plausible message and nothing behind it, the same shape as gates 22, 25 and 28. The verdict deliberately stays FAIL, because nothing here can tell a crash from a differently-phrased finding and guessing wiring would turn a real parity failure into a green hole. It simply stops claiming a count nobody measured. gate-61 carries the same clamp idiom and it is left alone with the reason written down: its wiring branch sits immediately above, so a crash cannot reach it, and zero counted lines with a non-zero exit is format drift rather than a dead checker. Two suites updated, neither weakened. The crashed-checker suite now accepts gate-17's NOT APPLICABLE on its E2BIG arm only when the terminal count marker proves the interpreter ran over the oversized scope list, which is a stronger assertion than the two it sits beside; its verdict regex also gained the multi-word form that made NOT APPLICABLE parse as NOT. The route-auth suite's gate-14 expectation moves from PASS to NOT APPLICABLE for the same reason its gate-5 neighbour moved in August: a pass here is scoping, not absence, and that reclassification was only half done. Refs: .github#374, #330, #364, #379

* feat(gates)!: scan the whole tree by default — ADR-020 superseded The gates now scan the ENTIRE tracked codebase. Diff scoping becomes opt-in via --scope-to-diff / --diff / HYDRA_GATE_SCOPE=diff. Ruben's rationale: when a gate changes or is added, the next push to beta should fail unless the old code is brought up to the new standard, so gate updates travel with releases instead of applying only to code written after them. Recorded in hydra-gates/ADR-020-SUPERSEDED.md, including what it costs: the first development-to-beta run after this lands surfaces the fleet backlog at once, and the last wide-scope measurement was roughly 3,900 findings. The scope is now two independent, named inputs rather than one. File scope controls which files the state gates open and defaults to full; the delta base is resolved separately and is what the five delta gates compare against. Gates 16, 29, 47, 48 and 61 are keyed on whether a base was resolved, not on the file scope, so a pull request keeps every delta gate and gains whole-tree state coverage. Without a base they report NOT APPLICABLE by name, never PASS. An unresolvable base stays fatal on a diff-scoped run and is non-fatal at full scope, where it costs five gates rather than all sixty-four. BASE_REF no longer carries a hardcoded origin/development default, which was the shared root cause of the gate-16 and gate-61 empty-scope defects. Eighteen gates could report PASS over a scope they never opened. Full scope removes the empty set in the common case but leaves the fall-through latent, so it is fixed at the fall-through through one shared helper: gates 14, 17, 18, 20, 21, 22, 34 through 44 and 52. Gates 14 and 20 were found by sweeping the table and are not in the issue's list of sixteen. Gate 40's inline note arguing against this was false on both of its claims and is deleted with the fix; gate 52's uncomputed ratchet half and gate 53's uncomputed orphan-promotion half now say so on stdout instead of hiding behind a PASS. APP_DIR is absolutised before the cd and the resolved absolute path is printed. Gate 17 was the only gate handing APP_DIR to its checker afterwards, so a relative app-dir made the scan root resolve a second time against the app dir itself: same tree, absolute path FAIL 1, relative path PASS. CI was never exposed because bin/hydra-gates absolutises; the documented human invocation was. The acceptance suite now enforces the empty-scope property across the package instead of for seven named gates, and carries a relative-path arm the standing driver is structurally unable to express because it always builds absolute paths. Both new arms were proven able to fail by deliberate breakage. Helper suites 74 passed / 2 quarantined / 1 failed before and after, the failure being the pre-existing unresolvable-ajv needs-external skip that fails closed locally and is green in CI. Entry-point tests 61 passed / 0 failed before, 69 passed / 0 failed after. Refs: .github#374, #361, #347, #364, #371 * docs(gates): the runner header described the old default The Options block still said --scope-to-diff was the mode and named origin/development as the base default. Both are gone: full scope is the default, and BASE_REF starts empty because that hardcoded default is the shared root cause of .github#347 and #361. * fix(gates): a crashed gate-61 read as an empty scope, and gate-24 invented a count Two repairs, both from the same rule: a crashed checker must never read as an empty scope, and an empty scope must never read as a pass. gate-61 — the first draft of the delta-base change skipped the checker invocation entirely when no base was resolved and short-circuited to the empty-scope status. Nothing ran, so nothing could crash, and test_gate_crashed_checker_is_not_a_finding.sh caught it immediately: with a python3 that cannot run, gate-61 reported NOT APPLICABLE where it must report SKIPPED (wiring). That is the same regression #364's first draft made in gate-16, caught by the same suite, and the repair is the same one: always run the checker, evaluate wiring first and scope second. With no base the helper is now invoked in its whole-tree mode purely to learn whether it can run, and its findings are discarded — sweeping on a baseless run was tried before and reverted because the builder runs that way. gate-24 — the parity wrapper is app-owned and declares no terminal marker, so a wrapper that exited non-zero having written nothing parseable was reported as one parity violation: a fabricated number with a plausible message and nothing behind it, the same shape as gates 22, 25 and 28. The verdict deliberately stays FAIL, because nothing here can tell a crash from a differently-phrased finding and guessing wiring would turn a real parity failure into a green hole. It simply stops claiming a count nobody measured. gate-61 carries the same clamp idiom and it is left alone with the reason written down: its wiring branch sits immediately above, so a crash cannot reach it, and zero counted lines with a non-zero exit is format drift rather than a dead checker. Two suites updated, neither weakened. The crashed-checker suite now accepts gate-17's NOT APPLICABLE on its E2BIG arm only when the terminal count marker proves the interpreter ran over the oversized scope list, which is a stronger assertion than the two it sits beside; its verdict regex also gained the multi-word form that made NOT APPLICABLE parse as NOT. The route-auth suite's gate-14 expectation moves from PASS to NOT APPLICABLE for the same reason its gate-5 neighbour moved in August: a pass here is scoping, not absence, and that reclassification was only half done. Refs: .github#374, #330, #364, #379 * fix(gate-61): the scope-matrix invariant read the FILE scope, not whether a diff existed The gate-agnostic property — a NOT APPLICABLE may not blame a diff on a run that computed none — is exactly right and stays. Its TRIGGER was wrong: it keyed on --full, when the property is about whether a diff was COMPUTED. Those are two independent inputs now, so the suite was reading the wrong one. That is the same conflation .github#347 was, one layer up, inside the test written to catch it. It fired on this branch and it fired correctly. Arm 3 now removes the BASE rather than the file scope, so every assertion it already carried holds verbatim. Arm 3b covers what used to be unreachable and is now what every PR in the fleet runs: full file scope with a real base. There, naming the diff is a true statement, and what has to hold instead is the falsifiability condition whose absence made #347 unreadable for weeks — a reason may name a diff only on a run that PRINTED the base it diffed against, so a reader can check the claim rather than trust it. Asserted from both sides. The advisory whole-tree sweep now runs in BOTH empty-scope branches. It ran only on the no-base path, because that used to be the only way to reach a state where the size of the unread backlog mattered. Full scope is the default now, so "a base resolved and excluded every registration" is the common case, and the ambiguity the sweep was added to remove had simply moved to the path everyone runs: 0 of 1 and 0 of 45 still printed identically there. The whole point of the reversal is that inherited debt stops being invisible, and leaving this gate's backlog unstated on the common path would have exempted it from exactly that. Refs: .github#374, #347 * ci(gates): the unresolvable-base check asserted the contract this PR supersedes The required "Unresolvable base ref must exit 99 with no green" step encoded ADR-020's rule as the ONLY acceptable behaviour, so this PR could not merge: under the new default an unresolvable base no longer aborts, it falls back to full scope. The old assertion is kept BYTE-FOR-BYTE and simply run in the mode it was always about (--scope-to-diff): with an explicit diff request and no usable base, the runner must still exit 99, print NOTHING WAS CHECKED, and emit no gate lines. Every clause of it still holds. A second step covers the new default: the run must say the base was unresolvable, must still emit gate lines, and none of gates 16/29/47/48/61 may print PASS over a scope that was never read. Positive-controlled against this runner's real captured output four ways — injecting "[gate-16] ... PASS" fails it, deleting a gate line fails it, and the diff-mode arm fails if the 99/NOTHING WAS CHECKED behaviour regresses.

rubenvdlinde merged commit 6b5baac into main Aug 10, 2026
31 checks passed

rubenvdlinde deleted the fix/gates-crashed-helper-must-not-judge branch August 10, 2026 10:38

Navigation Menu

Uh oh!

fix(gates): nine gates that turned a crashed helper into a verdict — 6, 7, 9 and 59-64 - #330

fix(gates): nine gates that turned a crashed helper into a verdict — 6, 7, 9 and 59-64#330
rubenvdlinde merged 1 commit into
mainfrom
fix/gates-crashed-helper-must-not-judge

rubenvdlinde commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rubenvdlinde commented Aug 10, 2026

Band 1 — the construct itself (gates 6, 7, 9)

Band 2 — the mirror image (gates 59, 60, 61, 62, 63, 64)

What is deliberately left tolerant

Proof, both directions

Regression coverage

Not in this PR

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant