Skip to content

ci: gate the two release-drift failures this release actually hit - #639

Merged
padak merged 2 commits into
mainfrom
claude/release-drift-gates
Aug 22, 2026
Merged

ci: gate the two release-drift failures this release actually hit#639
padak merged 2 commits into
mainfrom
claude/release-drift-gates

Conversation

@padak

@padak padak commented Aug 22, 2026

Copy link
Copy Markdown
Member

Stacked on #638 — review/merge that first.

Why

Preparing 0.88.0 surfaced two mistakes that no check in this repo could see. Both are mechanical, so both are now gated rather than left to the next person's diligence.

1. make version-gate-check (new, in CI)

Rejects any (since vX.Y.Z) / X.Y.Z+ marker whose version has no CHANGELOG entry.

keboola-expert.md turns those markers into a hard refusal rule — the agent compares the user's installed kbagent version and declines anything newer. A marker naming a version that never ships makes the agent refuse a flag the user actually has. Collapsing the unreleased 0.89.0 into 0.88.0 broke 20 markers across 8 files at once, and only a hand-grep caught them.

Verified it catches the real thing:

$ sed -i '' 's/(since v0.88.0)/(since v0.89.0)/' .../gotchas.md
$ make version-gate-check
ERROR: version gate names a version with no changelog entry.
  0.89.0 is not a CHANGELOG key -- named by:
    plugins/kbagent/skills/kbagent/references/gotchas.md:4103
    ... (5 locations)
exit=1

Scope is deliberately the two explicit gate syntaxes, never a bare version-looking string. Prose carries keboola-mcp-server v1.76.2, manifest v3, RFC 8628 — flagging those makes the check unusable. The cost is that a stale bare mention (# 0.89.0 MCP-parity flags in a test comment) passes silently; those mislead a reader, not the agent's version gate. Documented in the script docstring.

Offline (reads the tree only), so unlike changelog-check it runs per-PR in CI. That splits the existing CI comment's claim that these tags "need judgement": whether a marker names the right version does and stays with /kbagent:review; that the version exists does not.

Covers 418 markers across 71 versions today, all resolving.

2. make changelog-check gains the inverse direction

It verified every release has an entry. Nothing verified every entry has a release.

A version bump rides along with any PR touching pyproject.toml; publishing is a separate manual step. So a merge-train strands a whole block and the next tag absorbs it — notes written from one block, the other shipped unannounced. Exactly what happened here.

It has happened 18 times against 161 published releases:

0.47.0 0.47.2 0.48.0 0.51.1 0.60.1 0.60.2 0.60.3 0.60.4
0.63.3 0.63.4 0.67.0 0.68.0 0.69.0 0.70.0 0.70.1 0.78.0 0.81.0 0.83.0

Frozen in KNOWN_UNRELEASED as a baseline the check stops from growing — publishing those tags retroactively would advertise a wheel nobody can install, and rewriting old notes reaches nobody who already read them.

One correctness note worth reviewing: the audit is bounded to the window gh release list --limit actually returned. Without that floor, a truncated fetch reports every early release as unpublished — the first draft did exactly that (--limit 50 vs 173 changelog keys → 124 false positives). The same truncation is why the orphan count was first measured as 5 rather than 18; the limit is also raised 50 → 500 so the window covers the whole history.

Tests

20 added, all offline. Both helpers are pure, so the audits are unit-testable without network; the live-repo assertions guard the guards — test_the_scan_actually_finds_gates fails if a broken glob would make the version-gate check pass vacuously.

make check green: 5853 passed, 12 skipped.


Open in Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@padak
padak force-pushed the claude/release-drift-gates branch from fab8f63 to 39b1af5 Compare August 22, 2026 15:11
padak added 2 commits August 22, 2026 17:20
Preparing 0.88.0 surfaced two mistakes that no check in the repo could see.
Both are mechanical, so both are now gated.

1. Version gates naming a version that never ships
   `make version-gate-check` (new, and wired into the CI `check` job) rejects
   any `(since vX.Y.Z)` / `X.Y.Z+` marker whose version has no CHANGELOG entry.

   `keboola-expert.md` turns those markers into a hard refusal rule: the agent
   compares the user's installed version and declines anything newer. A marker
   naming a version that never ships therefore makes the agent refuse a flag
   the user actually has. Collapsing 0.89.0 into 0.88.0 broke 20 markers across
   8 files at once, and only a hand-grep caught them.

   Scope is the two explicit gate syntaxes, never a bare version-looking
   string -- prose carries `keboola-mcp-server v1.76.2`, `manifest v3`,
   `RFC 8628`, and flagging those would make the check unusable. Cost of that
   choice: a stale bare mention passes silently. It covers 418 markers across
   71 versions today, all resolving.

   Offline, unlike changelog-check, so it runs per-PR in CI. The CI comment
   claiming these tags "need judgement" is split: whether a marker names the
   RIGHT version does and stays with /kbagent:review; that the version EXISTS
   does not.

2. Changelog entries that were never published
   `make changelog-check` gains the inverse of the audit it already ran. It
   verified every release has an entry; nothing verified every entry has a
   release. A version bump rides along with any PR touching pyproject.toml
   while publishing is a separate manual step, so a merge-train strands a whole
   block and the next tag absorbs it -- its notes written from one block, the
   other shipped unannounced. That is what happened here.

   It has happened 18 times against 161 published releases. Those are frozen in
   KNOWN_UNRELEASED as a baseline the check exists to stop growing; publishing
   a tag retroactively would only advertise a wheel nobody can install. The
   audit is bounded to the window `gh release list --limit` actually returned,
   so a truncated fetch cannot report early releases as missing -- the first
   draft of this check did exactly that, and the same truncation is why the
   orphan count was first reported as 5.

20 tests added. Both helpers are pure so they are unit-testable without
network; the live-repo assertions guard the guards (a broken glob would make
the version-gate check pass vacuously).
The gate job's comment claims it runs the same silent-drift gates ci.yml
enforces so a tag pushed directly cannot bypass them. The version-gate check
was added to ci.yml in the parent commit but not here, which left exactly the
wrong hole: a renumbered release is a mistake that reaches a TAG rather than a
PR, and it breaks every marker naming the dropped version at once.
@padak
padak force-pushed the claude/release-drift-gates branch from 39b1af5 to 670da35 Compare August 22, 2026 15:23
@padak
padak merged commit fb6aa96 into main Aug 22, 2026
4 checks passed
@padak
padak deleted the claude/release-drift-gates branch August 22, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant