Skip to content

test(gate-16): the fixture #364 shipped without - #366

Merged
rubenvdlinde merged 1 commit into
mainfrom
test/gate-16-scope-fixture
Aug 11, 2026
Merged

test(gate-16): the fixture #364 shipped without#366
rubenvdlinde merged 1 commit into
mainfrom
test/gate-16-scope-fixture

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

#364 (merged as 765d081) changed a gate verdict fleet-wide and shipped with no
fixture behind it. This is the fixture, driven through the real bin/hydra-gates
over a purpose-built two-commit repository, and observed RED before the change
and GREEN after
:

tree result
81c8c97 (pre-#364) 3 failed / 9 passed — arm 3: ".github#361 is LIVE: gate-16 printed PASS on a --full run that computed no diff, over a tree whose two untagged methods the positive control just named"
765d0817 (post-#364) 0 failed / 12 passed

Both of #364's load-bearing claims were re-derived, not taken on trust

Each by patching the runner and re-running this same fixture:

counterfactual observed
_skip 16 … scope instead of na [gate-16] spec-coverage: FAIL — internal error: _skip called with reason category 'scope' — a fleet-wide false RED, exactly as #364 states. The suite now asserts against it by name.
a full sweep on --full (the gate-19 else-branch shape) [gate-16] FAIL — 2 changed method(s) missing @spec, naming LegacyDebtController, which no author touched. ADR-020 violated.

⚠️ One refinement to the second claim, for the record. A literal copy of
gate-19's else-branch — merely dropping HYDRA_GATE_BASE_REF= — would not have
swept at all: check_spec_coverage.py:508 defaults the env lookup to
origin/development, so it returns # count=0 and changes nothing. The
false-RED danger is real, but it arrives one line later, with the
check_visual_coverage.py or None precedent. Nothing in #364 is wrong; the
mechanism is one step removed from where the PR body places it.

The arms

All through bin/hydra-gates, one tree, only the scope input changed:

positive control  --mode report NAMES LegacyDebtController, uncovered_count=2
arm 1  diff, docs only          PASS      (ADR-020: inherited debt is not mine)
arm 2  diff ADDS untagged work  FAIL, and NAMES NewWorkController.php
arm 3  --full, arm 1's tree     NOT APPLICABLE, reason names the ABSENCE of a
                                diff, no legacy finding in the log at all,
                                gate-16 listed in the coverage summary's NA block

Arms 1 and 2 are what make arm 3 readable: without them a NOT APPLICABLE is
ambiguous between "gate-16 is honest about an empty scope" and "this fixture
contains nothing to find".

Arm 3 asserts the absence of LegacyDebtController from the gate's log, not
just the absence of a FAIL verdict — so a full sweep cannot creep back in wearing
a NOT APPLICABLE.

Coverage ratchet

gate-16 moves from UNCOVERED.md (needs-diff, whose stated reason — "it needs a
purpose-built two-commit history, not a fixture tree" — was accurate and is now
satisfied) to COVERED-ELSEWHERE.md. 12 of 64 fixtured → 13 of 64.

Verified green on origin/main at 765d081: test_gate16_spec_coverage_scope.sh
12/12, test_gate_acceptance_matrix.sh 4/4, test_gate_crashed_checker_is_not_a_finding.sh
ALL PASS, ShellCheck clean (koalaman/shellcheck:stable against the repo's own
.shellcheckrc).

#364 changed a verdict with no fixture behind it. This adds one, driven through
the real bin/hydra-gates over a purpose-built two-commit repository, and observed
RED before the change and GREEN after:

  before (81c8c97)  3 failed / 9 passed  -- arm 3: "#361 is LIVE: gate-16 printed
                                            PASS on a --full run that computed no
                                            diff, over a tree whose two untagged
                                            methods the positive control just named"
  after  (274b926)  0 failed / 12 passed

Both of #364's load-bearing claims were re-derived here rather than taken on
trust, by patching the runner and re-running this same fixture:

  * category `scope` instead of `na`  -> [gate-16] FAIL - internal error:
    _skip called with reason category 'scope'. A fleet-wide false RED, exactly
    as the PR body says. The suite now asserts against it by name.
  * a full sweep on --full           -> [gate-16] FAIL - 2 changed method(s)
    missing @SPEC, naming LegacyDebtController, which no author touched. ADR-020
    violated. The suite also asserts the legacy surface never reaches the log,
    so a sweep cannot creep back in wearing a NOT APPLICABLE.

REFINEMENT to the second claim, for the record: a LITERAL copy of gate-19's
else-branch (dropping HYDRA_GATE_BASE_REF=) would not have swept at all --
check_spec_coverage.py:508 defaults the env lookup to 'origin/development', so
it returns `# count=0` and changes nothing. The false-RED danger is real but
arrives one line later, with the check_visual_coverage.py `or None` precedent.

The arms, all through bin/hydra-gates on one tree:

  positive control  --mode report names LegacyDebtController, 2 uncovered
  arm 1  diff, docs only          PASS      (ADR-020: inherited debt is not mine)
  arm 2  diff ADDS untagged work  FAIL, and NAMES NewWorkController.php
  arm 3  --full, arm 1's tree     NOT APPLICABLE, reason names the ABSENCE of a
                                  diff, no legacy finding in the log, gate-16
                                  listed in the coverage summary's NA block

Coverage ratchet: gate-16 moves from UNCOVERED.md (`needs-diff`) to
COVERED-ELSEWHERE.md. 12 of 64 fixtured -> 13 of 64.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant