build(changelog): harden and improve the AI changelog generator (#338)

[devondragon]

Summary

Reworks generate_changelog.py (invoked during ./gradlew release via the generateAIChangelog task wired into beforeReleaseBuild) so it can never abort a release and produces output much closer to our hand-written CHANGELOG voice.

Closes #338.

What changed

Reliability

• AI generation and the git-collection phase are wrapped so any failure (missing OPENAI_API_TOKEN, network/API error, git error, decode error) degrades to a deterministic offline generator or skips — it never aborts beforeReleaseBuild.
• Fails fast (exit 2) when no version is passed on a non-interactive shell instead of hanging on input().
• Model/temperature are configurable (OPENAI_MODEL, OPENAI_TEMPERATURE); retries gracefully if a model rejects a custom temperature. Stale "GPT-4o" comment removed.

CHANGELOG.md handling

• New section is inserted below the # Changelog title (Keep a Changelog style), an existing same-version section is de-duplicated, and an ## [Unreleased] section is consumed as an authoritative draft and reset. This resolves the release-time duplication we clean up by hand today.

Prompt quality

• Leads with a ### Security category (CWE / SUF-xx IDs), adds ### Behavior changes (client impact) and a SemVer classification + rationale, gives repo context (library, user.* config, numeric response code), constrains to exact class/property/endpoint names, and uses the most recent existing CHANGELOG section as a few-shot style exemplar.
• Feeds commit bodies and referenced PR descriptions (best-effort via gh), not just subjects.

Cost / security

• Overall prompt-size budget (CHANGELOG_PROMPT_CHAR_BUDGET) with graceful degradation (drop diffs, then truncate).
• The diff-to-OpenAI data-sharing is now documented in the module docstring, with CHANGELOG_NO_DIFFS (metadata only) and CHANGELOG_SKIP_AI (fully offline) opt-outs.

Categorization / cleanup

• Dropped the near-useless whole-diff substring heuristic; leftover commits are categorized by file paths (docs-only / tests-only). Release-plumbing commits ([Gradle Release Plugin], *-SNAPSHOT) are filtered.
• Consolidated two git show calls into one. import re is now used.
• Added requirements.txt (pinned openai) and a stdlib unittest suite (24 tests). Added --dry-run.

Config / env vars

OPENAI_API_TOKEN, OPENAI_MODEL (default gpt-5), OPENAI_TEMPERATURE (default 0.2), CHANGELOG_SKIP_AI, CHANGELOG_NO_DIFFS, CHANGELOG_PROMPT_CHAR_BUDGET (default 200000), CHANGELOG_FETCH_PR_BODIES.

Not done (by design)

The ticket floated a hard human-approval gate for the generated changelog. I did not add a blocking gate inside the automated release (it would break ./gradlew release). Instead: --dry-run, a "review and refine" marker on offline output, and preserve-and-refine of hand drafts. Happy to add an opt-in gate if you'd prefer.

Testing

• python -m unittest test_generate_changelog → 24 passing (categorization, plumbing filter, split/compose/update, Unreleased reset, same-version dedup, budget degradation, fallback, code-fence stripping, non-fatal collection failure, skip-AI does not call gh).
• Offline end-to-end: CHANGELOG_SKIP_AI=1 python generate_changelog.py 5.1.1 --dry-run against real history.
• Verified the write path against a copy of the real CHANGELOG.md: single title, inserted below title, no 5.1.0 duplication, order preserved.
• No OpenAI calls were made (no key used).

[Copilot]

Pull request overview

This PR substantially reworks the release-time generate_changelog.py script to be best-effort (never aborting ./gradlew release) while producing changelog output closer to the project’s desired voice and structure.

Changes:

• Hardened changelog generation with offline fallback, prompt budgeting, improved categorization, and safer git/AI/PR-body collection behavior.
• Improved CHANGELOG.md updating: insert below header area, consume/reset ## [Unreleased], and de-duplicate same-version entries.
• Added pinned Python dependency (openai) plus a stdlib unittest suite covering the new pure-function behavior.

Reviewed changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 4 comments.

File	Description
generate_changelog.py	Major reliability, prompt/categorization, budget controls, and Keep-a-Changelog write-path improvements for release-time changelog generation.
test_generate_changelog.py	New unittest suite validating categorization, changelog splitting/composition, budget degradation, and non-fatal behavior.
requirements.txt	Pins openai for reproducible release tooling installs.
.gitignore	Ignores common Python artifacts for the new release tooling/tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[claude]

Review: build(changelog): harden and improve the AI changelog generator (#338)

This is a well-structured rewrite of generate_changelog.py. The CHANGELOG.md split/compose logic, the file-based categorization fallback (dropping the old whole-diff substring heuristic), and the 24-test unittest suite covering the pure functions are all solid improvements. Since this is release tooling (not part of the published library), the Java-specific conventions in CLAUDE.md don't directly apply, but I evaluated it against general Python best practices and the PR's own stated reliability goal.

Correctness — reliability gap vs. the stated "never abort" goal

The PR description says: "AI generation and the git-collection phase are wrapped so any failure … degrades to a deterministic offline generator or skips — it never aborts beforeReleaseBuild."

Looking at build_changelog_body(), only the actual generate_ai_changelog(prompt) call is inside the try/except Exception block:

```python
fetch_pr_bodies(commits)
allow_diffs = not os.environ.get("CHANGELOG_NO_DIFFS")
context = build_context(categorized, allow_diffs=allow_diffs)
exemplar = read_exemplar()
prompt = build_prompt(context, exemplar, human_draft)
try:
return generate_ai_changelog(prompt)
except Exception as exc:
...
return generate_fallback_changelog(categorized, human_draft)
```

fetch_pr_bodies() shells out to gh pr view per referenced PR. Its inner helper _fetch_single_pr() only catches subprocess.SubprocessError/OSError — an unexpected failure mode (e.g. gh returning a JSON array instead of an object, a TypeError/AttributeError from malformed output, or some other edge case) would propagate out of fetch_pr_bodies, out of build_changelog_body, and out of main() uncaught, since main() also doesn't wrap this call:

```python
version = resolve_version(args.version)
body = build_changelog_body(commits, version) # not wrapped
```

That would crash the script with a non-zero exit and traceback — exactly the release-aborting failure mode this PR is meant to eliminate. read_exemplar() (a file read) and build_context/build_prompt are unlikely to fail but are similarly outside the safety net.

Suggestion: widen the try/except in build_changelog_body() to wrap everything from fetch_pr_bodies(commits) through generate_ai_changelog(prompt), so any part of the AI path (not just the API call itself) degrades to the fallback generator.

Test coverage

The 24 unit tests thoroughly cover the pure functions (categorization, split/compose, budget degradation, dedup, etc.), but there's no test exercising the "AI path throws → falls back to deterministic generator" behavior (e.g. mocking generate_ai_changelog to raise and asserting build_changelog_body still returns a fallback body). Given that graceful degradation is the headline feature of this PR, a test like test_ai_failure_falls_back_to_deterministic would directly validate the reliability contract — and would have caught the gap above.

Security consideration (minor, already partly addressed)

The module docstring is upfront about diffs being sent to OpenAI, and CHANGELOG_NO_DIFFS/CHANGELOG_SKIP_AI opt-outs exist — good transparency. Given this is explicitly called out as "a security-focused library," it might be worth considering whether diff-sharing should be opt-in rather than opt-on-by-default (i.e. flip the default to CHANGELOG_NO_DIFFS=1 unless explicitly enabled), so a contributor doesn't need to know about the env var to avoid leaking a diff that happens to contain something sensitive. Not blocking — just flagging since the PR itself frames this as a deliberate trade-off worth a second look.

Minor / nit

• RELEASE_PLUMBING_RE matches -SNAPSHOT anywhere in the subject line. A legitimate commit whose subject happens to mention a snapshot version (e.g. "chore: verify build against 6.0.0-SNAPSHOT") would be silently dropped from the changelog entirely rather than just deprioritized. Probably rare in practice, but worth a comment noting the trade-off if intentional.
• generate_fallback_changelog() filters out merge commits (MERGE_COMMIT_RE) before rendering, but _render_context() (which builds the AI prompt) does not apply the same filter — merge-commit noise still reaches the model. The prompt does instruct the model to "ignore release-plumbing noise," so this is likely fine, but the two code paths aren't quite symmetric.

Summary

Nice improvement overall — much more robust categorization, sensible prompt-budget degradation, and good test coverage of the deterministic logic. The main thing I'd fix before merging is closing the gap between the "AI failures never abort the release" claim and what's actually wrapped in try/except, since that's the core reliability promise of the PR.

[devondragon]

Thanks @copilot and Claude for the review — addressed in 3b37b20.

Fixed

• No-tags root commit omission (Copilot): get_last_reference() now returns None and get_commits() uses a HEAD range, so an initial release (no tags) no longer drops the root commit.
• Reliability gap in the try/except (Claude): widened the guard in build_changelog_body to cover fetch_pr_bodies / build_context / read_exemplar / build_prompt, and added an isinstance(dict) guard in _fetch_single_pr, so no AI-path failure can abort the release. Added a test_ai_failure_falls_back_to_deterministic test that validates the contract directly.
• Unbounded PR fetches (Copilot + Claude): added CHANGELOG_MAX_PR_FETCHES (default 50) capping the number of gh pr view calls.
• -SNAPSHOT over-matching (Claude): narrowed RELEASE_PLUMBING_RE so a legit commit mentioning a snapshot version isn't dropped; the Gradle Release Plugin's own messages still match.
• Unused local (Copilot): removed.

Intentionally not changed (with rationale)

• Flip diff-sharing to opt-in (CHANGELOG_NO_DIFFS=1 default) (Claude, non-blocking): keeping diffs on-by-default is a deliberate choice for output quality; the data-sharing is documented and there are two opt-outs (CHANGELOG_NO_DIFFS, CHANGELOG_SKIP_AI). Easy one-line flip if we later prefer the safer default.
• Filter merge commits from the AI context too (Claude, "likely fine"): the asymmetry is intentional — merge commits carry the PR numbers whose descriptions I fetch, so dropping them from the AI context would lose that PR-body context. Added a code comment documenting this.

Test suite is now 28 passing (python -m unittest test_generate_changelog).

[claude]

Review

This is a well-scoped rewrite of generate_changelog.py, the release-tooling script (not part of the library's runtime code, so no impact on consumers). Overall the change is a clear improvement: better reliability guarantees, structured git collection, dedup/consumption of ## [Unreleased], a documented data-sharing trade-off, and 24 new unittest tests. Nice work.

Strengths

• Reliability contract is well-implemented: main() wraps git collection in try/except -> return 0, and build_changelog_body() wraps the entire AI path (PR fetch, context build, API call) in try/except -> deterministic fallback. Backed by direct tests (test_main_returns_zero_on_collection_failure, test_ai_failure_falls_back_to_deterministic).
• compose_changelog/split_changelog correctly handle title-preservation, Unreleased-section consumption, and same-version dedup - and each behavior has a targeted test.
• fetch_pr_bodies correctly bounds actual gh calls via the cache dict even though it iterates over all commits/PR numbers (test_pr_fetch_capped confirms this).
• Good security disclosure: the module docstring is upfront about diffs going to OpenAI, with CHANGELOG_SKIP_AI / CHANGELOG_NO_DIFFS opt-outs. No secrets (API tokens) are ever printed/logged.
• Subprocess calls use argument lists (no shell=True), so no command-injection risk from commit text or PR numbers.

Potential issues

1. Categorization heuristic uses loose prefix matching, not exact-word matching (generate_changelog.py, categorize_commits). Prefixes are checked with str.startswith(("doc", "docs")), ("test", "tests")), etc. against subject.split(":", 1)[0]. This causes false-positive matches for real conventional-commit-style prefixes that merely share a leading substring:

   • "docker: bump image" -> prefix "docker" -> startswith("doc") is True -> miscategorized as Documentation.
   • "testcontainers: bump version" -> prefix "testcontainers" -> startswith("test") is True -> miscategorized as Testing.

   Since this repo uses Testcontainers heavily, a real testcontainers:-prefixed commit is plausible. Consider anchoring the match, e.g. prefix in ("doc", "docs") or a regex requiring an exact-word match, instead of startswith. Low severity since this only affects the pre-categorization heuristic (the AI prompt already tells the model to trust the actual changes over the heuristic categories), but it does directly affect the offline/deterministic fallback output, which has no such correction step.

2. Module-level env parsing isn't guarded (generate_changelog.py:77-78): PROMPT_CHAR_BUDGET = int(os.environ.get("CHANGELOG_PROMPT_CHAR_BUDGET", "200000")) and MAX_PR_FETCHES = int(...) run at import time, outside any try/except. A typo'd non-numeric value for either env var raises ValueError before main() even starts, which would abort the whole script - undermining the PR's core "never abort a release" goal for this one failure mode. Worth wrapping in a small helper that falls back to the default on a bad value, given how central "never crash the release" is to this PR's purpose.

3. _create_completion's temperature-retry detection is string-matching based (generate_changelog.py:562-573): it decides whether to retry without temperature by checking "temperature" in str(exc).lower(). This is a reasonable best-effort approach (and won't cause incorrect behavior since it still raises for any other exception), but it's a bit fragile if OpenAI ever changes error message wording - just flagging as a minor robustness note, not a blocker.

Test coverage
Coverage is strong for the pure functions (categorization, split/compose, dedup, budget degradation, fallback behavior, PR-fetch capping, reliability contracts). Nothing directly exercises the module-level env-var parsing issue in point 2 above - worth a quick regression test if that's addressed (e.g. CHANGELOG_PROMPT_CHAR_BUDGET=abc shouldn't crash).

Minor / non-blocking

• requirements.txt pins openai==1.105.0 but the script also depends on the gh CLI being present for PR-body fetching; that's already handled gracefully (shutil.which("gh") check), just noting it's an implicit environment dependency worth a one-line mention in docs if not already covered elsewhere.
• The PR description already flags that a hard human-approval gate was intentionally not added - that's a reasonable call given --dry-run and the preserve/refine-of-drafts behavior.

Nothing here is blocking; the reliability guarantees and dedup logic (the core goals of this PR) are correctly implemented and well tested. The two categorization/env-parsing points are low-severity edge cases worth a follow-up rather than a required fix before merge.