Skip to content

Stop review rounds from compounding on their own earlier findings - #40

Open
gobetti wants to merge 1 commit into
mainfrom
review-round-scope-and-convergence
Open

gobetti wants to merge 1 commit into
mainfrom
review-round-scope-and-convergence

Conversation

@gobetti

@gobetti gobetti commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Problem

When a PR got a review round, addressed it, and pushed a fix, the next round treated that fix as fresh review surface — raising new non-blocking suggestions about the very tests, comments, and guards the previous round had asked for. Repeated, it leaves an author chasing rounds that never converge.

A recent ht-platform PR ran 13 review rounds that produced 0 criticals and 1 warning between them, alongside ~28 suggestions. The last six rounds consisted entirely of findings about code that existed only because of earlier rounds: a guard added in one round, a test for that guard in the next, the accuracy of that test's assertions after that, then the wording of the comment describing the test. The author stopped manually rather than converging.

The prompt had no counterweight. There was no test for whether a finding was in scope, no floor under what was worth reporting, and a tie-break (if you are unsure whether something is a warning or suggestion, it is a suggestion) that routed every unconfirmed item into the output as a confident-looking recommendation.

Change

Adds a # Reporting rules section to the base prompt, placed after inputs.extra_prompt so it gets the last word, with an explicit precedence line over anything above it:

  • Scope test every finding must pass: introduced or worsened by this PR, reachable with today's types and data, and consequential. A comment, docblock, or test name that this PR leaves stale counts as worsened — fixing it belongs in the PR that broke it.
  • Confidence, separated from severity. Suggestion now asserts the reviewer is confident and that this PR is the better place to fix it. Anything unconfirmed goes under Worth checking as a question the author can close by answering. Uncertainty never downgrades into a Suggestion.
  • A reporting floor listing what is never a finding (comment phrasing that isn't actually wrong, test-internal hygiene with no failure mode, coverage parity with an equally uncovered precedent, hardening against unreachable states), plus a capped Follow-ups section for real but out-of-scope items.
  • A round header and convergence line. use_sticky_comment: true means each round overwrites the last, so the escalation was invisible in the PR — reconstructing the 13 rounds above required reading the sticky comment's edit history via GraphQL.
  • A ## Later rounds rule: remediation commits get verified, not rediscovered. An item is not reportable if it exists only because an earlier round asked for a change, unless it is a Critical or Warning on its own merits.

Scope

Prompt text only — additive, no new inputs, no caller changes, so every consuming repo picks it up with no edits.

Deliberately left out of this first unit, pending a trial: a mechanical review_mode input with a hard-excluded regression-only prompt block, and the corresponding caller-side changes in ht-platform (which still instructs a consistency pass over fix commits). The ## Later rounds rules carry that restraint prompt-side for now and explicitly override it; if that proves too weak in practice, the mechanical version is the escalation.

Testing

None automated — this is prompt text, and its effect only shows on real review rounds. Worth watching on the next few multi-round PRs: whether Later rounds actually beats the caller-side instruction it conflicts with, and whether round numbers stay coherent enough to read the escalation at a glance.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


📝 Auto-filled

When a PR got a review round, addressed it, and pushed a fix, the next
round treated that fix as fresh review surface and raised new
non-blocking suggestions about the very tests, comments, and guards the
previous round had asked for. Repeated, it leaves an author chasing
rounds that never converge. One recent PR ran 13 rounds that produced 0
criticals and 1 warning, the last six consisting entirely of findings
about code that existed only because of earlier rounds.

The prompt had no counterweight: no test for whether a finding was in
scope, no floor under what was worth reporting, and a tie-break sending
every unconfirmed item into the output as a Suggestion.

Adds a reporting-rules section after the caller's extra_prompt, which it
explicitly overrides:

- a scope test every finding must pass (introduced or worsened here,
  reachable, consequential) — a comment or test name this PR leaves
  stale counts as worsened, so fixing it belongs here
- Suggestion now means confident and better fixed in this PR; anything
  unconfirmed goes under Worth checking as a question to the author,
  and uncertainty never downgrades into a Suggestion
- a floor listing what is never reportable, plus a capped Follow-ups
  section for real but out-of-scope items
- a round header and a convergence line, since the sticky comment
  overwrites each round and hides the escalation
- a later-rounds rule: remediation commits get verified, not
  rediscovered

Prompt-only and additive. No new inputs and no caller changes, so every
consuming repo picks it up as-is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants