fix(gate-28): an EMPTY DIFF SCOPE is not a structural gap — #172 swung one state too far - #180
Conversation
…g one state too far #172 was right that gate-28 must not report PASS having opened zero files. But there are TWO ways to read zero files and it gave them the same word: (a) lib/**/*.php files ARE in this diff and none carries a licence tag — a genuine gap in the repository, correctly `structural`; (b) no lib/**/*.php file is in this diff AT ALL — diff-scoping working exactly as ADR-020 designed it, and no statement about the repo. (b) is the ordinary case. `_in_scope` filters to the PR's diff, so every workflow-only and frontend-only PR reads zero lib PHP files. Calling that `structural` counts it against coverage, and with hydra-gates-require-full-coverage on by default the run fails with exit 98 for a licence problem the repository does not have. MEASURED on the fleet's own unpinning PRs, each a single-file workflow diff: hrmq#74 CLEAN baseline -> RED on nothing but gate-28 app-versions#129 gate 28 its only gate that did not run opencatalogi#813 " nextcloud-app-template#132 " launchpad#60 " hrmq's 168 lib PHP files all carry their tags. The gate had simply not been handed any of them to read, and then reported that as the repository's fault. This is the "a gate that is legitimately not applicable must NOT fail the run" rule broken — the same shape as #173, one gate lower down. It surfaced now because the fleet just unpinned (#177): at v1.3.0 gate-28 reported PASS on an empty read, so the false GREEN hid what is now a false RED. Neither was a verdict. The `structural` branch is preserved for case (a) and now names the count of files that WERE in scope, so the two states no longer read alike. TESTS — scripts/lib/test_gate_license_triangle_scope.sh, four control pairs, discovered automatically by tests/run-helper-suites.sh: wf-only diff, tagged repo -> NOT APPLICABLE (b, the false red) untagged PHP in the diff -> structural (a, must stay red) tagged PHP in the diff -> PASS (the gate compared something) no lib/ at all -> NOT APPLICABLE (#172's original case) Verified in BOTH directions rather than assumed: against the fixed runner 5/5 green; against the runner at 954a78a the first two FAIL and the last three still pass, so the suite discriminates and the fix is targeted rather than a mute for #172. The did-not-run assertion took two attempts to measure anything — v1 matched only a summary line this suite never triggers, v2 matched a roster the runner also prints for NOT-APPLICABLE gates. Both are recorded in the file, because a dead assertion inside a suite meant to tell two states apart is the exact defect the suite exists to catch.
… one assertion twice ShellCheck SC2015 on the new suite, and it is not a style note here. In `grep -q ... && _ok "$d" || _bad "$d"`, a non-zero return from _ok makes _bad run as well, so the SAME assertion is counted once as a pass and once as a failure. In a harness whose entire job is to count assertions correctly, that is instrument corruption — the class this suite exists to catch, one level up. Also adds a `*)` arm: an unknown expectation string now fails loudly instead of matching no case and silently asserting nothing. Re-verified in both directions after the change: fixed runner 5 passed / 0 failed, runner at 954a78a 3 passed / 2 failed, entry-point invariants 39/0.
…f-scope-is-not-structural # Conflicts: # hydra-gates/scripts/run-hydra-gates.sh
|
Rebased onto While this was open, What is not superseded is the 234-line control suite, which Proven able to fail, not merely observed passing:
The sabotage failure names the consequence precisely:
That is the exact regression the PO ruled out ("if a gate is legitimately not applicable the flag shouldnt fail the run"), so it is worth having a test that fails when it returns. |
What this is
Found while unpinning the fleet (#177). It is a blocker for that rollout, and it is the same shape as #173 one gate lower down.
The defect
#172 was right that gate-28 must not report
PASShaving opened zero files. But there are two ways to read zero files, and it gave them the same word:structural— a real gapNOT APPLICABLE— diff-scoped out per ADR-020_in_scopefilters to the PR's diff, so (b) is the ordinary case: every workflow-only and frontend-only PR reads zero lib PHP files. Calling thatstructuralcounts it against coverage, and withhydra-gates-require-full-coverageon by default the run fails withexit 98for a licence problem the repository does not have.Measured, not theorised
On the fleet's own unpinning PRs — each a single-file workflow diff:
hrmq's 168 lib PHP files all carry their tags. The gate had simply not been handed any of them to read, and then reported that as the repository's fault.It surfaced now because the fleet just unpinned: at
v1.3.0gate-28 reportedPASSon an empty read, so the false GREEN was hiding what is now a false RED. Neither was a verdict.The fix
The
structuralbranch is preserved for case (a), and now names the count of files that were in scope so the two states no longer read alike.Tests
scripts/lib/test_gate_license_triangle_scope.sh— four control pairs, discovered automatically bytests/run-helper-suites.sh:Verified in both directions rather than assumed. Against the fixed runner 5/5 green. Against the runner at
954a78athe first two assertions FAIL and the last three still pass — so the suite discriminates, and the fix is targeted rather than a mute for #172.The did-not-run assertion took two attempts to measure anything (v1 matched only a summary line this suite never triggers; v2 matched a roster the runner also prints for NOT-APPLICABLE gates). Both attempts are documented in the file, because a dead assertion inside a suite meant to tell two states apart is the exact defect the suite exists to catch.
Refs #172, #173, #177