Skip to content

feat: framework_sources -- record which framework version a mapping was made against - #247

Merged
chaksaray merged 1 commit into
developfrom
feat/framework-sources
Sep 3, 2026
Merged

chaksaray merged 1 commit into
developfrom
feat/framework-sources

Conversation

@chaksaray

Copy link
Copy Markdown
Contributor

Closes #245. Originating case: OWASP/www-project-mcp-top-10#52, credit to @Santoshkumarpuppala for the observation this closes a gap on: a commit-pinned number-to-slug map fixes future readings and does nothing for records already emitted — the pin has to travel in the record.

What this adds

framework_sources, an optional object keyed by mapping field (owasp_mcp, owasp_asi, mitre_atlas, nist_ai_rmf), each entry carrying version/commit/read_date, or a pin_status: unpinnable declaration with read_date and unpinnable_reason for frameworks with nothing to pin against — the MCP Top 10's actual current state. Reuses the exact vocabulary already established for crosswalk endpoint pinning (schema/crosswalk-1.0.0.schema.json) rather than inventing a second one for the same problem.

Real counts checked before designing the shape:

owasp_mcp:   80 of 80 records carry this field
owasp_asi:   69 of 80 records carry this field
mitre_atlas: 50 of 80 records carry this field
nist_ai_rmf: 56 of 80 records carry this field

Option B (one container field, keyed by mapping name) over Option A (four parallel sibling fields, precise but verbose) or Option C (corpus-level manifest, wrong for the actual problem — records are mapped at different times against different readings, which is exactly how #52's divergence happened; a single corpus-wide statement can't represent that).

Schema fields added to both schema/ave-record-1.1.0.schema.json (used by validate_records.py) and schema/ave-record.schema.json (used by scripts/build-records.js) — confirmed identical before and after, kept in sync per the existing convention.

The check, and a deliberate deviation from the usual CI pattern

scripts/check_framework_sources.py follows check_vulnerability_taxonomy.py's soft-warn/--strict/--only shape. It is not wired into CI in this PR, and CONTRIBUTING.md gets only the --strict --only new-record gate line, not a corpus-wide soft-warn line.

Why: run unscoped against the live corpus, it reports all 80 records missing a framework_sources entry for at least one carried mapping — 255 individual field findings in one line. #242 (still open) already raised the soft-warning-channel-becoming-noise concern after taking check_confidence_signal.py's CI output from one line to nine; this field would be meaningfully noisier than that on day one, before any backfill judgment has happened. Corpus-wide CI enforcement is deferred to the separate backfill task (real per-record judgment on what each was actually mapped against — "unknown, assigned before anyone tracked this" is a legitimate answer the schema allows), not bundled into this schema PR.

Validated

  • python scripts/validate_records.py: 80/80 still valid (only pre-existing, unrelated researcher-attribution warnings)
  • python scripts/check_fixtures.py: all pass
  • python -m pytest tests/ -x -q: 446 passed (435 existing + 11 new)
  • node scripts/build-records.js: builds clean against the updated ave-record.schema.json

Every new test mutation-checked by hand: reverted the read_date requirement in both the pinned and unpinnable branches, the strict/warn exit-code, and the --only filter — each caused exactly the tests naming that behavior to fail, no more (one mutation, the per-field "does this record carry this mapping" guard, correctly failed most of the suite, since nearly every test depends on that guard existing).

dist/ intentionally left untouched — regenerating it only bumped generated_at, no real content changed since no record uses the new optional field yet.

…as made against (closes #245)

Originating case: OWASP/www-project-mcp-top-10#52. Credit to
Santoshkumarpuppala for the observation this issue is built on: a
commit-pinned number-to-slug map fixes future readings and does
nothing for records already emitted -- the pin has to travel in the
record itself.

Adds framework_sources as an optional object keyed by mapping field
(owasp_mcp, owasp_asi, mitre_atlas, nist_ai_rmf), each entry carrying
version/commit/read_date, or a pin_status: unpinnable declaration with
read_date and unpinnable_reason for frameworks with no release to pin
against (the MCP Top 10's current state). Reuses the same vocabulary
already established for crosswalk endpoint pinning
(schema/crosswalk-1.0.0.schema.json) rather than inventing a second
one for the same problem.

Option B (one container field, keyed by mapping field) over Option A
(four parallel sibling fields) or Option C (corpus-level manifest):
frameworks version differently (MCP Top 10 has no release, MITRE
ATLAS versions discretely, NIST AI RMF is a dated publication) and
records are mapped at different times against different readings,
which is exactly how the divergence in #52 happened -- a corpus-level
statement can't represent that.

Schema fields added to both schema/ave-record-1.1.0.schema.json (used
by validate_records.py) and schema/ave-record.schema.json (used by
scripts/build-records.js), kept in sync per the existing convention.

scripts/check_framework_sources.py follows check_vulnerability_taxonomy.py's
soft-warn/--strict/--only shape, with one deliberate difference: it is
NOT wired into CI in this PR, and CONTRIBUTING.md only gets the
--strict --only new-record gate line, not the corpus-wide soft-warn
line. Run unscoped against the live corpus it reports all 80 records
missing a framework_sources entry for one or more of their carried
mappings (255 individual field findings) -- meaningfully noisier than
the '#242 took one line to nine' volume warning that prompted this
caution. Corpus-wide CI enforcement is deferred to the separate
backfill task (per-record judgment on what each was actually mapped
against, including 'unknown' as a legitimate answer), not bundled into
this schema PR.

Every new test mutation-checked by hand: reverted the read_date
requirement in both the pinned and unpinnable branches, the strict/warn
exit-code, and the --only filter -- each caused exactly the tests that
name that behavior to fail, and no others beyond the one structural
guard (the per-field 'does this record even carry this mapping' check)
that most of the suite legitimately depends on.
@chaksaray
chaksaray merged commit c35afa9 into develop Sep 3, 2026
6 checks passed
@chaksaray
chaksaray deleted the feat/framework-sources branch September 3, 2026 00:17
chaksaray added a commit that referenced this pull request Sep 3, 2026
chaksaray added a commit that referenced this pull request Sep 3, 2026
…as made against- #247 (#252)

Co-authored-by: chaksaray <15962335+chaksaray@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record which framework version a mapping was made against (framework_sources)

1 participant