feat: framework_sources backfill (owasp_asi, mitre_atlas), and a schema fix it needed first - #256
Merged
Merged
Conversation
…ma fix it needed first Closes #255. ## The schema fix pin_status only had "unpinnable" -- declaring a framework has no version/tag/commit at all to pin against. That's not what most of the corpus needed to say. MITRE ATLAS ships monthly releases, the OWASP Top 10 for Agentic Applications prints "Version 2026" on its own cover (re-fetched fresh this session to confirm, not from memory), NIST AI RMF is a dated publication -- none of them are actually unpinnable. What's missing is a way to say "this framework has real versions, but which one this record's tag was checked against predates any tracking of it, and isn't recoverable now." Added pin_status: "unknown" for exactly that. Same shape as unpinnable (needs read_date, no version/commit), but unpinnable_reason doesn't apply -- the gap is in AVE's own recordkeeping, not a property of the framework. Additive to both schema/ave-record-1.1.0.schema.json and schema/ave-record.schema.json (kept identical, per the existing convention), plus scripts/check_framework_sources.py's has_real_source() and two new mutation-checked tests mirroring the existing unpinnable ones. ## The backfill, real archaeology only Every date/version below is from a git commit, a merged PR, or a primary source fetched fresh this session -- nothing reconstructed from memory. - owasp_asi, 69 of 69 currently-tagged records: the 2026-08-23 audit (#196 + #199, merged same day) verified every then-existing tagged record against the primary-source PDF. framework_sources.owasp_asi = {version: "2026", read_date: "2026-08-23"} for all 69. - mitre_atlas, 40 of 50 currently-tagged records: issue #127 (merged as #164, 2026-08-09) scored 49 records citing T0043/T0048/T0051/T0054 against ATLAS.yaml fetched live that day; 40 of those still carry a mitre_atlas tag today (the other 9 had it dropped entirely). MITRE's real release history shows v2026.07 published 2026-08-07, two days before the audit -- very likely what was fetched, but not contemporaneously recorded, so this goes in as pin_status: "unknown" + read_date: "2026-08-09" rather than an asserted version. Deferred, not silently skipped (documented in #255 with reasons): mitre_atlas on the other 10 currently-tagged records (each has only its own record-creation commit, no dedicated verification event), nist_ai_rmf on all 56 tagged records (no audit trail found anywhere), and owasp_mcp on all 80 records (the actual live-divergence case from OWASP/www-project-mcp-top-10#52 -- deserves its own dedicated, per-record pass against a specific MCP Top 10 commit, not a rushed bulk fill here). ## Validated - python scripts/validate_records.py: 80/80 valid - python scripts/check_fixtures.py: all pass - pytest tests/ -x -q: 463 passed (461 + 2 new) - node scripts/build-records.js: dist/ regenerated, diff is only the intended additions plus the expected generated_at bump - Every touched record diffed by hand: minimal, byte-exact insertions after derivable_into, no unrelated reformatting
This was referenced Sep 4, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #255.
The schema fix
pin_statusonly had"unpinnable"— declaring a framework has no version/tag/commit at all to pin against. That's not what most of the corpus needed to say. MITRE ATLAS ships monthly releases, the OWASP Top 10 for Agentic Applications prints "Version 2026" on its own cover (re-fetched fresh this session to confirm, not from memory), NIST AI RMF is a dated publication — none of them are actually unpinnable. What's missing is a way to say "this framework has real versions, but which one this record's tag was checked against predates any tracking of it, and isn't recoverable now."Added
pin_status: "unknown"for exactly that. Same shape asunpinnable(needsread_date, no version/commit), butunpinnable_reasondoesn't apply — the gap is in AVE's own recordkeeping, not a property of the framework. Additive to bothschema/ave-record-1.1.0.schema.jsonandschema/ave-record.schema.json(kept identical, per the existing convention), plusscripts/check_framework_sources.py'shas_real_source()and two new mutation-checked tests mirroring the existingunpinnableones.The backfill, real archaeology only
Every date/version below is from a git commit, a merged PR, or a primary source fetched fresh this session — nothing reconstructed from memory.
framework_sources.owasp_asi = {version: "2026", read_date: "2026-08-23"}for all 69.T0043/T0048/T0051/T0054againstATLAS.yamlfetched live that day; 40 of those still carry amitre_atlastag today (the other 9 had it dropped entirely). MITRE's real release history showsv2026.07published 2026-08-07, two days before the audit — very likely what was fetched, but not contemporaneously recorded, so this goes in aspin_status: "unknown"+read_date: "2026-08-09"rather than an asserted version.Deferred, not silently skipped (documented in #255 with reasons):
mitre_atlason the other 10 currently-tagged records (each has only its own record-creation commit, no dedicated verification event),nist_ai_rmfon all 56 tagged records (no audit trail found anywhere), andowasp_mcpon all 80 records (the actual live-divergence case fromOWASP/www-project-mcp-top-10#52— deserves its own dedicated, per-record pass against a specific MCP Top 10 commit, not a rushed bulk fill here).Validated
python scripts/validate_records.py: 80/80 validpython scripts/check_fixtures.py: all passpytest tests/ -x -q: 463 passed (461 + 2 new)node scripts/build-records.js:dist/regenerated, diff is only the intended additions plus the expectedgenerated_atbumpderivable_into, no unrelated reformatting (an earlier draft of this backfill script round-tripped every record throughjson.dump, which silently unescaped unicode and collapsed single-element arrays across the whole corpus — caught before committing, redone as a precise text-level insertion instead)