Skip to content

fix(gate-28): an EMPTY DIFF SCOPE is not a structural gap — #172 swung one state too far - #180

Merged
rubenvdlinde merged 3 commits into
mainfrom
fix/gate-28-empty-diff-scope-is-not-structural
Aug 6, 2026
Merged

fix(gate-28): an EMPTY DIFF SCOPE is not a structural gap — #172 swung one state too far#180
rubenvdlinde merged 3 commits into
mainfrom
fix/gate-28-empty-diff-scope-is-not-structural

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

What this is

Found while unpinning the fleet (#177). It is a blocker for that rollout, and it is the same shape as #173 one gate lower down.

The defect

#172 was right that gate-28 must not report PASS having opened zero files. But there are two ways to read zero files, and it gave them the same word:

state correct verdict
(a) lib PHP files are in this diff, none carries a licence tag structural — a real gap
(b) no lib PHP file is in this diff at all NOT APPLICABLE — diff-scoped out per ADR-020

_in_scope filters to the PR's diff, so (b) is the ordinary case: every workflow-only and frontend-only PR reads zero lib PHP files. Calling that structural counts it against coverage, and with hydra-gates-require-full-coverage on by default the run fails with exit 98 for a licence problem the repository does not have.

Measured, not theorised

On the fleet's own unpinning PRs — each a single-file workflow diff:

repo baseline with @main
hrmq#74 CLEAN RED on nothing but gate-28
app-versions#129 gates red gate 28 its only gate that did not run
opencatalogi#813 gates green gate-28
nextcloud-app-template#132 gates red gate-28
launchpad#60 gates red gate-28

hrmq's 168 lib PHP files all carry their tags. The gate had simply not been handed any of them to read, and then reported that as the repository's fault.

It surfaced now because the fleet just unpinned: at v1.3.0 gate-28 reported PASS on an empty read, so the false GREEN was hiding what is now a false RED. Neither was a verdict.

The fix

The structural branch is preserved for case (a), and now names the count of files that were in scope so the two states no longer read alike.

Tests

scripts/lib/test_gate_license_triangle_scope.sh — four control pairs, discovered automatically by tests/run-helper-suites.sh:

wf-only diff, tagged repo  -> NOT APPLICABLE  (b, the false red)
untagged PHP in the diff   -> structural      (a, must stay red)
tagged PHP in the diff     -> PASS            (the gate compared something)
no lib/ at all             -> NOT APPLICABLE  (#172's original case)

Verified in both directions rather than assumed. Against the fixed runner 5/5 green. Against the runner at 954a78a the first two assertions FAIL and the last three still pass — so the suite discriminates, and the fix is targeted rather than a mute for #172.

The did-not-run assertion took two attempts to measure anything (v1 matched only a summary line this suite never triggers; v2 matched a roster the runner also prints for NOT-APPLICABLE gates). Both attempts are documented in the file, because a dead assertion inside a suite meant to tell two states apart is the exact defect the suite exists to catch.

Refs #172, #173, #177

Conduction Release Bot added 3 commits August 6, 2026 08:01
…g one state too far

#172 was right that gate-28 must not report PASS having opened zero files.
But there are TWO ways to read zero files and it gave them the same word:

  (a) lib/**/*.php files ARE in this diff and none carries a licence tag
      — a genuine gap in the repository, correctly `structural`;
  (b) no lib/**/*.php file is in this diff AT ALL — diff-scoping working
      exactly as ADR-020 designed it, and no statement about the repo.

(b) is the ordinary case. `_in_scope` filters to the PR's diff, so every
workflow-only and frontend-only PR reads zero lib PHP files. Calling that
`structural` counts it against coverage, and with
hydra-gates-require-full-coverage on by default the run fails with exit 98
for a licence problem the repository does not have.

MEASURED on the fleet's own unpinning PRs, each a single-file workflow diff:

  hrmq#74                     CLEAN baseline -> RED on nothing but gate-28
  app-versions#129            gate 28 its only gate that did not run
  opencatalogi#813            "
  nextcloud-app-template#132  "
  launchpad#60                "

hrmq's 168 lib PHP files all carry their tags. The gate had simply not been
handed any of them to read, and then reported that as the repository's
fault. This is the "a gate that is legitimately not applicable must NOT
fail the run" rule broken — the same shape as #173, one gate lower down.

It surfaced now because the fleet just unpinned (#177): at v1.3.0 gate-28
reported PASS on an empty read, so the false GREEN hid what is now a false
RED. Neither was a verdict.

The `structural` branch is preserved for case (a) and now names the count
of files that WERE in scope, so the two states no longer read alike.

TESTS — scripts/lib/test_gate_license_triangle_scope.sh, four control pairs,
discovered automatically by tests/run-helper-suites.sh:

  wf-only diff, tagged repo  -> NOT APPLICABLE  (b, the false red)
  untagged PHP in the diff   -> structural      (a, must stay red)
  tagged PHP in the diff     -> PASS            (the gate compared something)
  no lib/ at all             -> NOT APPLICABLE  (#172's original case)

Verified in BOTH directions rather than assumed: against the fixed runner
5/5 green; against the runner at 954a78a the first two FAIL and the last
three still pass, so the suite discriminates and the fix is targeted rather
than a mute for #172.

The did-not-run assertion took two attempts to measure anything — v1 matched
only a summary line this suite never triggers, v2 matched a roster the
runner also prints for NOT-APPLICABLE gates. Both are recorded in the file,
because a dead assertion inside a suite meant to tell two states apart is
the exact defect the suite exists to catch.
… one assertion twice

ShellCheck SC2015 on the new suite, and it is not a style note here. In
`grep -q ... && _ok "$d" || _bad "$d"`, a non-zero return from _ok makes
_bad run as well, so the SAME assertion is counted once as a pass and once
as a failure. In a harness whose entire job is to count assertions
correctly, that is instrument corruption — the class this suite exists to
catch, one level up.

Also adds a `*)` arm: an unknown expectation string now fails loudly
instead of matching no case and silently asserting nothing.

Re-verified in both directions after the change: fixed runner 5 passed /
0 failed, runner at 954a78a 3 passed / 2 failed, entry-point invariants
39/0.
…f-scope-is-not-structural

# Conflicts:
#	hydra-gates/scripts/run-hydra-gates.sh
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

Rebased onto main and narrowed to the test only.

While this was open, main landed the same behavioural fix — and in a more complete form: it splits the empty-scope case into two distinct na messages (diff-scoped-out vs. lib/ with no tracked PHP), and keeps both structural paths for the genuine gap. So the runner change here was superseded. I took main's implementation wholesale; run-hydra-gates.sh on this branch is now byte-identical to main.

What is not superseded is the 234-line control suite, which main does not have. That is all this PR now contains.

Proven able to fail, not merely observed passing:

state exit result
main's implementation 0 5 passed / 0 failed
empty-scope na reverted to structural 1 3 passed / 2 failed
restored 0 5 passed / 0 failed

The sabotage failure names the consequence precisely:

FAIL — gate-28 was named in GATES THAT DID NOT RUN despite being NOT APPLICABLE — it will still fail the run under --require-full-coverage

That is the exact regression the PO ruled out ("if a gate is legitimately not applicable the flag shouldnt fail the run"), so it is worth having a test that fails when it returns.

@rubenvdlinde
rubenvdlinde merged commit 787647d into main Aug 6, 2026
27 checks passed
@rubenvdlinde
rubenvdlinde deleted the fix/gate-28-empty-diff-scope-is-not-structural branch August 6, 2026 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant