Review and improve generate_changelog.py (AI changelog generator run during release)

[devondragon]

Goal

Review generate_changelog.py and improve it. The script is invoked automatically during ./gradlew release via the generateAIChangelog task (build.gradle:266-268, wired into beforeReleaseBuild at line 277). It collects commits since the last tag, pre-categorizes them, sends the commit messages + diffs to the OpenAI API, and prepends the generated Markdown to CHANGELOG.md. Because it runs inside the release flow, a failure or bad output directly affects releases.

This is a review-and-improve task, not a reported outage. Below are concrete observations from a first read to seed the review — confirm/reject each and implement what's worthwhile.

Correctness / reliability

• Unhandled failure aborts the release. generate_changelog() (lines 169-181) makes an OpenAI call with no try/except. A missing OPENAI_API_TOKEN, network error, or API error throws mid-beforeReleaseBuild, potentially leaving a release half-done. Make the changelog step fail gracefully (skip/warn) or clearly non-fatal.
• Stale/misleading model config. Line 173 comment says "Use GPT-4o (without fallback)" but line 175 sets model="gpt-5". Confirm the intended model and fix the comment. Consider making model/temperature configurable rather than hardcoded.
• Interactive input() fallback can hang CI. Lines 220-224 fall back to input() when no version arg is passed. The release task passes the version, but any other invocation path in a non-interactive shell would hang. Consider failing fast instead.
• Diff-based categorization is near-useless. categorize_commits() fallback (lines 87-92) does substring matching for fix/bug/add/new/implement against the entire diff text. Almost every diff contains these words, so commits without a conventional prefix get bucketed arbitrarily. Either drop the diff-heuristic or make it meaningfully stronger.

CHANGELOG.md handling

• Prepends above everything, including any title. update_changelog() (lines 187-199) writes the new ## [version] block at byte 0 of the file, ahead of any # Changelog header/preamble. This likely contributes to the known release-time changelog duplication / "section above the title" behavior we clean up manually post-release. Insert after the title and consider adopting Keep a Changelog structure ([Unreleased], version links, dates).

Cost / context / security

• No total prompt-size budget. Per-commit diffs are truncated at 500 lines (lines 116-118) but there's no cap on the combined prompt. A release with many/large commits can blow the model context window or run up cost. Add an overall budget/summarization step.
• Full code diffs are sent to an external API. For a security-focused library, sending raw diffs to OpenAI is worth an explicit decision: document the data-sharing, and consider redaction or a commit-message-only mode. Diffs can contain secrets or sensitive logic.

Cleanup / minor

• import re (line 6) is unused.
• Two git show invocations per commit (--stat --patch at 38-40 and --name-status at 43-45) could be consolidated to reduce subprocess overhead.
• No dependency pinning (requirements.txt / locked openai version) and no tests. Consider adding both.
• Generated changelog is committed unreviewed during automated release. Consider a review/edit gate, or generating to a draft file for human approval.

Prompt & manual-content improvements (output quality)

Motivating evidence: in the current CHANGELOG.md, the ## [5.1.0] block appears twice — the raw script output sits above the # Changelog title (generic ### Features / Fixes / Breaking / Refactoring / Documentation / Testing / Other skeleton), and a hand-rewritten ## [5.1.0] sits below the title. The hand-written version is far better: a ### Security category, CWE IDs, SUF-01..06 references, a dedicated "Behavior changes (client impact)" section, HTTP status / response-code specifics, and a SemVer rationale. Today the AI output is effectively discarded and rewritten by hand. Goal: get the raw output close enough that it's edited, not replaced.

Prompt changes (prompt block at lines 126-167)

• Add a ### Security category and instruct the model to lead with it, referencing CWE IDs and ticket IDs (e.g. SUF-xx) found in commit text. The current format block (lines 144-164) has no Security section, yet it's the first thing added by hand for this library.
• Few-shot with a real hand-written entry. Embed a prior polished section (e.g. the current 5.1.0 or 5.0.1 block) as a "match this voice, structure, and altitude" exemplar. Likely the single biggest quality lever.
• Give the model repo context: this is a library, not an app; consumers configure via the user.* prefix; endpoints return a numeric response code; frame changes for consuming applications.
• Request a "Behavior changes (client impact)" subsection covering HTTP status / response-code changes and required consumer actions.
• Request a SemVer classification + rationale (major/minor/patch and why).
• Constrain names and drop noise: use exact class/property/endpoint names from the diffs (no invented names); ignore release-plumbing commits ([Gradle Release Plugin] …, *-SNAPSHOT bumps) that currently land in "Other." Set a low temperature for consistency.

Incorporate manually-written content (ranked by value-to-effort)

• Feed commit bodies, not just subjects (cheapest, high value). Line 24 uses --pretty=format:%H|%s — subject only. The detailed rationale (and squash-merged PR descriptions) lives in the body (%b) and merge-commit messages, which are dropped today. Adding %b hands the model prose already written by hand.
• Pull PR descriptions via gh. Merge commits carry PR numbers; fetching PR bodies surfaces the CWE analysis / client-impact writeups that usually originate there.
• Adopt a Keep-a-Changelog ## [Unreleased] section. Humans jot notes under [Unreleased] during the cycle; at release the script refines and merges those notes with commit analysis rather than generating from scratch, so hand-notes become authoritative input.
• Preserve-and-refine instead of blind prepend. If a hand-written section for the target version already exists, pass it to the model as "here's the human draft — complete/improve it, don't discard it," and insert below the title (also resolves the duplication cleaned up post-release).

Out of scope

Not proposing to remove the AI changelog approach — just to harden it. Keep the existing generateAIChangelog integration point.

[create-issue-branch]

Branch issue-338-Review-and-improve-generate_changelog-py-AI-changelog-generator-run-during-release created!

[devondragon]

PR ready for review: #340 (Closes #338 on merge).

[devondragon]

Work Complete — PR Merged

PR: build(changelog): harden and improve the AI changelog generator (#338)
Merged into: main (squash commit 6053467)
Branch: feature/338-improve-generate-changelog (deleted)

Summary of changes

• Reworked generate_changelog.py to be best-effort: AI-generation and git-collection failures degrade to a deterministic offline generator (or skip) instead of aborting ./gradlew release.
• Fixed CHANGELOG write path — insert below the title (Keep a Changelog), same-version de-dup, [Unreleased] consume/reset — resolving the release-time duplication.
• Reworked the prompt (Security/Behavior-changes/SemVer sections, few-shot exemplar, commit bodies + PR descriptions, exact-name constraints) and added a prompt-size budget with graceful degradation.
• Added requirements.txt (pinned openai), a 28-test stdlib unittest suite, --dry-run, and env opt-outs (CHANGELOG_SKIP_AI, CHANGELOG_NO_DIFFS, CHANGELOG_MAX_PR_FETCHES). Documented the diff-to-OpenAI data-sharing.

Files changed

• generate_changelog.py
• test_generate_changelog.py (new)
• requirements.txt (new)
• .gitignore

Note: a hard human-approval gate inside the automated release was intentionally not added (it would block ./gradlew release); addressed via --dry-run + preserve-and-refine of hand drafts instead.