Skip to content

feat: framework_sources backfill (owasp_asi, mitre_atlas), and a schema fix it needed first - #256

Merged
chaksaray merged 1 commit into
developfrom
feat/framework-sources-backfill
Sep 4, 2026
Merged

chaksaray merged 1 commit into
developfrom
feat/framework-sources-backfill

Conversation

@chaksaray

Copy link
Copy Markdown
Contributor

Closes #255.

The schema fix

pin_status only had "unpinnable" — declaring a framework has no version/tag/commit at all to pin against. That's not what most of the corpus needed to say. MITRE ATLAS ships monthly releases, the OWASP Top 10 for Agentic Applications prints "Version 2026" on its own cover (re-fetched fresh this session to confirm, not from memory), NIST AI RMF is a dated publication — none of them are actually unpinnable. What's missing is a way to say "this framework has real versions, but which one this record's tag was checked against predates any tracking of it, and isn't recoverable now."

Added pin_status: "unknown" for exactly that. Same shape as unpinnable (needs read_date, no version/commit), but unpinnable_reason doesn't apply — the gap is in AVE's own recordkeeping, not a property of the framework. Additive to both schema/ave-record-1.1.0.schema.json and schema/ave-record.schema.json (kept identical, per the existing convention), plus scripts/check_framework_sources.py's has_real_source() and two new mutation-checked tests mirroring the existing unpinnable ones.

The backfill, real archaeology only

Every date/version below is from a git commit, a merged PR, or a primary source fetched fresh this session — nothing reconstructed from memory.

Deferred, not silently skipped (documented in #255 with reasons): mitre_atlas on the other 10 currently-tagged records (each has only its own record-creation commit, no dedicated verification event), nist_ai_rmf on all 56 tagged records (no audit trail found anywhere), and owasp_mcp on all 80 records (the actual live-divergence case from OWASP/www-project-mcp-top-10#52 — deserves its own dedicated, per-record pass against a specific MCP Top 10 commit, not a rushed bulk fill here).

Validated

  • python scripts/validate_records.py: 80/80 valid
  • python scripts/check_fixtures.py: all pass
  • pytest tests/ -x -q: 463 passed (461 + 2 new)
  • node scripts/build-records.js: dist/ regenerated, diff is only the intended additions plus the expected generated_at bump
  • Every touched record diffed by hand: minimal, byte-exact insertions after derivable_into, no unrelated reformatting (an earlier draft of this backfill script round-tripped every record through json.dump, which silently unescaped unicode and collapsed single-element arrays across the whole corpus — caught before committing, redone as a precise text-level insertion instead)

…ma fix it needed first

Closes #255.

## The schema fix

pin_status only had "unpinnable" -- declaring a framework has no
version/tag/commit at all to pin against. That's not what most of the
corpus needed to say. MITRE ATLAS ships monthly releases, the OWASP Top
10 for Agentic Applications prints "Version 2026" on its own cover
(re-fetched fresh this session to confirm, not from memory), NIST AI
RMF is a dated publication -- none of them are actually unpinnable.
What's missing is a way to say "this framework has real versions, but
which one this record's tag was checked against predates any tracking
of it, and isn't recoverable now."

Added pin_status: "unknown" for exactly that. Same shape as
unpinnable (needs read_date, no version/commit), but unpinnable_reason
doesn't apply -- the gap is in AVE's own recordkeeping, not a property
of the framework. Additive to both schema/ave-record-1.1.0.schema.json
and schema/ave-record.schema.json (kept identical, per the existing
convention), plus scripts/check_framework_sources.py's
has_real_source() and two new mutation-checked tests mirroring the
existing unpinnable ones.

## The backfill, real archaeology only

Every date/version below is from a git commit, a merged PR, or a
primary source fetched fresh this session -- nothing reconstructed
from memory.

- owasp_asi, 69 of 69 currently-tagged records: the 2026-08-23 audit
  (#196 + #199, merged same day) verified every then-existing tagged
  record against the primary-source PDF. framework_sources.owasp_asi =
  {version: "2026", read_date: "2026-08-23"} for all 69.
- mitre_atlas, 40 of 50 currently-tagged records: issue #127 (merged as
  #164, 2026-08-09) scored 49 records citing T0043/T0048/T0051/T0054
  against ATLAS.yaml fetched live that day; 40 of those still carry a
  mitre_atlas tag today (the other 9 had it dropped entirely).
  MITRE's real release history shows v2026.07 published 2026-08-07,
  two days before the audit -- very likely what was fetched, but not
  contemporaneously recorded, so this goes in as pin_status: "unknown"
  + read_date: "2026-08-09" rather than an asserted version.

Deferred, not silently skipped (documented in #255 with reasons):
mitre_atlas on the other 10 currently-tagged records (each has only
its own record-creation commit, no dedicated verification event),
nist_ai_rmf on all 56 tagged records (no audit trail found anywhere),
and owasp_mcp on all 80 records (the actual live-divergence case from
OWASP/www-project-mcp-top-10#52 -- deserves its own dedicated,
per-record pass against a specific MCP Top 10 commit, not a rushed
bulk fill here).

## Validated

- python scripts/validate_records.py: 80/80 valid
- python scripts/check_fixtures.py: all pass
- pytest tests/ -x -q: 463 passed (461 + 2 new)
- node scripts/build-records.js: dist/ regenerated, diff is only the
  intended additions plus the expected generated_at bump
- Every touched record diffed by hand: minimal, byte-exact insertions
  after derivable_into, no unrelated reformatting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

framework_sources backfill, and a real gap found: pin_status has no 'unknown' state

1 participant