Add per-article BCQuality coverage for code-review dataset (AB#646153)

[Wenjie Fan (gggdttt)]

What

Adds per-article BCQuality coverage tracking to the code-review category so we can see, per BCQuality knowledge article, which articles have gold coverage and which have none. Follow-up from the 2026-08-06 PR Review Agent sync (AB#646153).

Why

Today gold entries are classified only by coarse domain/area. We cannot tell which specific BCQuality article an entry exercises, nor which articles have zero gold coverage. This blocks Wael's self-improving-agent pipeline from targeting coverage gaps.

Changes

• Schema (optional, backward-compatible) in dataset/codereview.py:
  • ArticleId — shared <domain>/<slug> constrained alias (e.g. security/secrettext-for-credentials).
  • ReviewComment.article — the article a required finding derives from.
  • CodeReviewEntryMetadata.articles — entry-level list for false-positive-guard entries (expected_comments = []) that test an article by omission and so have no per-comment article. Category-specific metadata lives on this EntryMetadata subclass rather than the shared base.
  • A model validator keeps the two annotation sources disjoint, and declared_articles() returns their union.
  • Existing entries without annotations load unchanged (counted as unannotated).
• analysis/bcquality_article_coverage.py — pure analysis logic: aggregate declared articles, enumerate the BCQuality article inventory from a checkout, compute covered / zero-coverage / unknown-slug / unannotated. Kept out of dataset, which stays the schema/model layer.
• notebooks/code-review-coverage.ipynb — the coverage report surface, mirroring the existing notebooks/dataset.ipynb. Inventory comes from $BCQUALITY_ROOT; when absent it degrades to declared-only (zero-coverage undetermined). Rendered outputs are committed so the analysis is readable directly on GitHub.
• Backfill: the security domain is annotated as a worked example (17 of its 20 entries → 9 distinct articles). Remaining domains are intentionally left for follow-up backfill; the report surfaces them as unannotated.

Example output (committed in the notebook)

9 articles covered across 17/130 annotated entries; 241 of 250 articles have zero coverage

Domain	Covered	Gold entries	Inventory	Zero-cov
security	9	17	24	15
performance	0	0	47	47
style	0	0	35	35
ui	0	0	29	29
...

The low overall ratio is expected at this stage: this PR delivers the mechanism plus a single worked domain, so 8 of 9 dataset areas are still unannotated by design.

Validation

ruff format/check clean (including the notebook), ty clean (the one remaining diagnostic is a pre-existing unused-ignore in redteam.py from main, untouched here), pytest 738 passed / 2 skipped. Dataset still loads all 130 entries; the JSONL diff touches only the security lines. Notebook executed end-to-end against BCQuality main, with non-deterministic execution timestamps stripped so the committed outputs stay reviewable.

Notes

• Additive — no existing scoring/behavior changes.
• No CLI subcommand: this is ad-hoc analysis, so it lives in notebooks/ per review feedback.
• Ties into the neutral ignored_comments work (Add neutral ignored_comments bucket to code-review scoring #768): FP-guard gold cases can be classified by the article whose false positive they guard.

AB#646153

[Copilot]

Pull request overview

Adds per-article BCQuality coverage tracking for code-review datasets.

Changes:

• Extends dataset schemas with optional article annotations.
• Adds inventory comparison and coverage reporting.
• Adds a CLI command, tests, and security-domain annotations.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
dataset/codereview.jsonl	Backfills security article annotations.
src/bcbench/commands/dataset.py	Adds the coverage CLI command.
src/bcbench/dataset/__init__.py	Exports coverage APIs.
src/bcbench/dataset/codereview.py	Adds comment-level article support.
src/bcbench/dataset/coverage.py	Implements coverage aggregation and inventory comparison.
src/bcbench/dataset/dataset_entry.py	Adds entry-level article metadata.
tests/test_codereview_coverage.py	Tests coverage logic and inventory handling.

[Sun Haoran (haoranpb)]

Wenjie Fan (@gggdttt) Quick question, is this function for ad-hoc analysis only? Or would you need it for downstream consumption?

[Copilot]

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (2)

notebooks/code-review-coverage.ipynb:68

• This sums article-entry associations rather than unique gold entries. If one entry declares two articles in the same domain, it is counted twice in the “Gold entries” column. Deduplicate the covered entry_ids before taking the count.

    "    row = {\"Domain\": domain, \"Covered\": len(covered), \"Gold entries\": sum(c.count for c in covered)}\n",

src/bcbench/dataset/coverage.py:106

• The advertised bcbench dataset coverage entry point is missing. src/bcbench/commands/dataset.py still registers only list, view, and version and never imports these helpers, so users get “No such command” and cannot use the documented --bcquality-root or --github-output options. Please register the coverage command and cover its console/JSON output paths.

def build_coverage_report(
    entries: Sequence[CodeReviewEntry],
    inventory: Iterable[str] | None = None,
) -> CoverageReport:

[Sun Haoran (haoranpb)]

Also noticed the notebook has no commited outputs, you can probably include that as well.

[Wenjie Fan (gggdttt)]

Committed the executed notebook outputs in b0a0e91. The rendered report now shows 9 covered articles across 17 of 130 annotated entries, a 186-article BCQuality inventory with 177 zero-coverage articles, per-domain totals, covered article-to-entry mappings, and the current gap lists. Execution metadata was normalized so the committed notebook contains no timestamps, local checkout paths, credentials, or transient errors.

[Sun Haoran (haoranpb)]

Have you read through the results in the notebook, just wondering if that coverage is by design, looks a bit surprising tbh.

9 articles covered across 17/130 annotated entries; 177 of 186 articles have zero coverage

Otherwise looks good