feat: framework_sources -- record which framework version a mapping was made against - #247
Merged
Merged
Conversation
…as made against (closes #245) Originating case: OWASP/www-project-mcp-top-10#52. Credit to Santoshkumarpuppala for the observation this issue is built on: a commit-pinned number-to-slug map fixes future readings and does nothing for records already emitted -- the pin has to travel in the record itself. Adds framework_sources as an optional object keyed by mapping field (owasp_mcp, owasp_asi, mitre_atlas, nist_ai_rmf), each entry carrying version/commit/read_date, or a pin_status: unpinnable declaration with read_date and unpinnable_reason for frameworks with no release to pin against (the MCP Top 10's current state). Reuses the same vocabulary already established for crosswalk endpoint pinning (schema/crosswalk-1.0.0.schema.json) rather than inventing a second one for the same problem. Option B (one container field, keyed by mapping field) over Option A (four parallel sibling fields) or Option C (corpus-level manifest): frameworks version differently (MCP Top 10 has no release, MITRE ATLAS versions discretely, NIST AI RMF is a dated publication) and records are mapped at different times against different readings, which is exactly how the divergence in #52 happened -- a corpus-level statement can't represent that. Schema fields added to both schema/ave-record-1.1.0.schema.json (used by validate_records.py) and schema/ave-record.schema.json (used by scripts/build-records.js), kept in sync per the existing convention. scripts/check_framework_sources.py follows check_vulnerability_taxonomy.py's soft-warn/--strict/--only shape, with one deliberate difference: it is NOT wired into CI in this PR, and CONTRIBUTING.md only gets the --strict --only new-record gate line, not the corpus-wide soft-warn line. Run unscoped against the live corpus it reports all 80 records missing a framework_sources entry for one or more of their carried mappings (255 individual field findings) -- meaningfully noisier than the '#242 took one line to nine' volume warning that prompted this caution. Corpus-wide CI enforcement is deferred to the separate backfill task (per-record judgment on what each was actually mapped against, including 'unknown' as a legitimate answer), not bundled into this schema PR. Every new test mutation-checked by hand: reverted the read_date requirement in both the pinned and unpinnable branches, the strict/warn exit-code, and the --only filter -- each caused exactly the tests that name that behavior to fail, and no others beyond the one structural guard (the per-field 'does this record even carry this mapping' check) that most of the suite legitimately depends on.
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #245. Originating case: OWASP/www-project-mcp-top-10#52, credit to @Santoshkumarpuppala for the observation this closes a gap on: a commit-pinned number-to-slug map fixes future readings and does nothing for records already emitted — the pin has to travel in the record.
What this adds
framework_sources, an optional object keyed by mapping field (owasp_mcp,owasp_asi,mitre_atlas,nist_ai_rmf), each entry carryingversion/commit/read_date, or apin_status: unpinnabledeclaration withread_dateandunpinnable_reasonfor frameworks with nothing to pin against — the MCP Top 10's actual current state. Reuses the exact vocabulary already established for crosswalk endpoint pinning (schema/crosswalk-1.0.0.schema.json) rather than inventing a second one for the same problem.Real counts checked before designing the shape:
Option B (one container field, keyed by mapping name) over Option A (four parallel sibling fields, precise but verbose) or Option C (corpus-level manifest, wrong for the actual problem — records are mapped at different times against different readings, which is exactly how #52's divergence happened; a single corpus-wide statement can't represent that).
Schema fields added to both
schema/ave-record-1.1.0.schema.json(used byvalidate_records.py) andschema/ave-record.schema.json(used byscripts/build-records.js) — confirmed identical before and after, kept in sync per the existing convention.The check, and a deliberate deviation from the usual CI pattern
scripts/check_framework_sources.pyfollowscheck_vulnerability_taxonomy.py's soft-warn/--strict/--onlyshape. It is not wired into CI in this PR, andCONTRIBUTING.mdgets only the--strict --onlynew-record gate line, not a corpus-wide soft-warn line.Why: run unscoped against the live corpus, it reports all 80 records missing a
framework_sourcesentry for at least one carried mapping — 255 individual field findings in one line. #242 (still open) already raised the soft-warning-channel-becoming-noise concern after takingcheck_confidence_signal.py's CI output from one line to nine; this field would be meaningfully noisier than that on day one, before any backfill judgment has happened. Corpus-wide CI enforcement is deferred to the separate backfill task (real per-record judgment on what each was actually mapped against — "unknown, assigned before anyone tracked this" is a legitimate answer the schema allows), not bundled into this schema PR.Validated
python scripts/validate_records.py: 80/80 still valid (only pre-existing, unrelated researcher-attribution warnings)python scripts/check_fixtures.py: all passpython -m pytest tests/ -x -q: 446 passed (435 existing + 11 new)node scripts/build-records.js: builds clean against the updatedave-record.schema.jsonEvery new test mutation-checked by hand: reverted the
read_daterequirement in both the pinned and unpinnable branches, the strict/warn exit-code, and the--onlyfilter — each caused exactly the tests naming that behavior to fail, no more (one mutation, the per-field "does this record carry this mapping" guard, correctly failed most of the suite, since nearly every test depends on that guard existing).dist/intentionally left untouched — regenerating it only bumpedgenerated_at, no real content changed since no record uses the new optional field yet.