Skip to content

Validate code-review article slugs against BCQuality - #804

Open
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
bench/validate-article-slugs
Open

Validate code-review article slugs against BCQuality#804
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
bench/validate-article-slugs

Conversation

@gggdttt

Copy link
Copy Markdown
Collaborator

Problem

ArticleId only validates the shape of an article reference:

^[a-z0-9][a-z0-9-]*/[a-z0-9][a-z0-9-]*$

Nothing checks that the article actually exists in BCQuality. A one-character typo passes schema validation and the entire test suite:

security/permission-set-avoid-wildcard-grants    schema=OK   exists_in_bcquality=True
security/permission-set-avoid-wildcard-grant     schema=OK   exists_in_bcquality=False

That is worse than a missing annotation, because it corrupts the coverage report in both directions: the entry is credited to a nonexistent article, and the real article silently stays in zero_coverage — so the very report meant to show which articles need gold answers hides a genuine gap.

Today the only signal is the unknown_articles field in notebooks/code-review-coverage.ipynb, which has to be run and read manually. With 17 of 144 entries annotated so far and ~215 comments still to tag, manual inspection does not scale.

Change

Add one opt-in dataset test to tests/test_bcquality_article_coverage.py:

  • Resolves a BCQuality checkout via the existing resolve_bcquality_root() (BCQUALITY_ROOT).
  • Loads the real dataset/codereview.jsonl and asserts no declared slug is missing from the enumerated inventory.
  • On failure, reports each unknown slug together with the entry ids that declare it, so the fix is immediately actionable.
  • Skips when BCQUALITY_ROOT is unset, so CI — which has no BCQuality checkout — is unaffected.

No production code changes; this reuses collect_declared_articles, enumerate_inventory and build_coverage_report as-is.

Validation

  • Injected a typo (...wildcard-grants -> ...wildcard-grant) into dataset/codereview.jsonl; the test failed with article slugs not found in <root>: {'security/permission-set-avoid-wildcard-grant': ['synthetic__security-005']}, then passed again after reverting.
  • With BCQUALITY_ROOT pointed at BCQuality main: 11 passed.
  • Without BCQUALITY_ROOT: 10 passed, 1 skipped.
  • Full suite: 738 passed, 3 skipped.
  • ruff format, ruff check clean; ty check reports only the pre-existing src/bcbench/redteam.py:150 unused-ignore warning.

The ArticleId pattern only checks the <domain>/<slug> shape, so a typo passes schema validation and every test. Such a typo is doubly wrong: it credits coverage to a nonexistent article and leaves the real one in zero-coverage.

Add an opt-in dataset test that resolves BCQUALITY_ROOT, enumerates the article inventory and fails on any declared slug missing from it, naming the offending entry ids. It skips when BCQUALITY_ROOT is unset, so CI (which has no BCQuality checkout) is unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant