Skip to content

Drop never-emitted comment key spellings from review parsing - #806

Open
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
bench/drop-dead-comment-keys
Open

Drop never-emitted comment key spellings from review parsing#806
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
bench/drop-dead-comment-keys

Conversation

@gggdttt

Copy link
Copy Markdown
Collaborator

What

_normalize_comment in evaluate/review_parsing.py accepted four key spellings that no producer emits:

Removed Kept, and why
path file (pr_review agent), filePath (BCApps review instructions)
lineEnd, endLine line_end (pr_review agent)
comment body (pr_review agent), issue (BCApps review instructions)

parse_review_output's bare-single-object check drops path for the same reason.

Why

The parser has exactly three known producers, and each was checked:

  • pr_review agent (agent/copilot/pr_review/review_output.py) emits file / line_start / line_end / body.
  • BCApps review instructions (agent/shared/instructions/microsoft-BCApps/instructions/*.md) ask for filePath / lineNumber / issue.
  • BCQuality review reports name the line field line, so that spelling is kept for a flattened report. Its path key belongs to references[], not to a finding's location, so it is not a file-key producer.

Across 488 finding objects in recorded agent outputs on disk, path, lineEnd, endLine and comment occur zero times. git log -S shows all four arrived in the original "Introducing Code Review category" commit, so they were speculative from the start rather than a fix for an observed parse failure. No test pinned them.

Severity aliases were audited in the same pass and deliberately left aloneblocker/major/minor/info come from BCQuality's report schema and error/warning/suggestion are parsed out of real reviewer-bot comment headers in collection/collect_codereview.py.

Validation

  • ruff format / ruff check clean; ty check shows only the pre-existing redteam.py diagnostic.
  • pytest 738 passed, 2 skipped.
  • Spot-checked the parser directly: pr_review, BCApps-instruction, flat-line and bare-single-object payloads all still parse to one comment each.
  • Existing tests already pin the surviving alternate spellings (test_codereview.py covers filePath / lineNumber / issue).

_normalize_comment accepted path/lineEnd/endLine/comment alongside the spellings that producers actually emit. None of the four has a producer: the pr_review agent emits file/line_start/line_end/body, the BCApps review instructions ask for filePath/lineNumber/issue, and BCQuality reports name the line field line. Across 488 findings in recorded agent outputs the four appear zero times, and they were introduced speculatively in the original code-review commit rather than in response to a parse failure.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 045bef88-d7d4-4553-a681-d7d73cdb2285
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant