Skip to content

ci: orphaned-fixture gate + CHANGELOG-parity gate (every fixtures/evals file consumed by a grader; every plugin.json bump requires a CHANGELOG entry) #663

Description

@kyle-sexton

Two process amendments from the 2026-07-20 merged-PR quality audit, both CI gates:

  1. Orphaned-fixture check: every file under a plugin's evals/fixtures/ (or equivalent) must be consumed by a grader — referenced in an eval files[] or asserted by a test. Prevents "looks-tested" ungraded suites (found: 16 ungraded autonomy security-binding fixtures, ~1,200 lines — see the sibling autonomy issue).
  2. CHANGELOG parity: a plugin.json version bump without a matching CHANGELOG.md entry fails the gate. The autonomy plugin has shipped 5 minor bumps with NO CHANGELOG.md at all while every other active plugin maintains one.

Placement note: repo-local gate script first (this repo's scripts/ + ci.yml, precedent #593/#609); if it stabilizes, promote to ci-workflows as reusable.

Third audit amendment — standardizing the golden-fixture idiom on #634's graded shape — is convention-shaped: routed to the decision queue rather than bundled here.


Reference: Boris Cherny, "Steps of AI Adoption" (Google Doc)

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions