You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to #245/#247 (the schema addition). Doing the backfill surfaced a
real design gap worth fixing before the backfill itself can be honest, plus
the actual archaeology for what's recoverable.
The gap
The schema as merged only allows pin_status: "unpinnable" — declaring
that a referenced framework "has no version, tag, or commit to pin
against." There is no way to express the very different, and far more
common, fact: the framework does have addressable versions, but which one
a given record's tag was originally checked against predates any tracking
of this at all.
Forcing unpinnable onto the bulk of the corpus to make the backfill
tractable would misrepresent frameworks that plainly do version — MITRE
ATLAS ships monthly releases (v2026.06, v2026.07, ...), the OWASP Top
10 for Agentic Applications prints "Version 2026" on its own cover page,
NIST AI RMF is a dated publication. None of them are actually unpinnable
in the sense the field currently only allows saying.
Proposed fix: add "unknown" as a second pin_status enum value,
meaning "the referenced framework has real versions; this record's
provenance predates provenance tracking and the specific version read is
not recoverable." Requires only read_date (same as unpinnable), no unpinnable_reason (the reason is structural and uniform — "predates
tracking" — not per-record). Additive: broadens what a valid record can
express, breaks nothing that currently validates.
Real archaeology, not reconstructed from memory
Every date/version below is from a git commit, a merged PR, an OWASP PDF
fetched fresh this session, or MITRE's own public release history —
checked directly, not recalled.
owasp_asi — 69 of 69 currently-tagged records, real dated event.
The 2026-08-23 audit (#196 + #199, merged same day) verified every
then-existing tagged record's owasp_asi value against the primary-source
PDF (genai.owasp.org/download/52117). Re-fetched that PDF just now to
confirm its own cover page reads "Version 2026," "December 2025" — framework_sources.owasp_asi = {"version": "2026", "read_date": "2026-08-23"} is fully real and verifiable for all 69.
mitre_atlas — 40 of 50 currently-tagged records, real dated event, no
recoverable exact version. Issue #127 (merged as #164, 2026-08-09)
scored 49 records citing T0043/T0048/T0051/T0054 against ATLAS.yaml fetched live that day. 40 of those 49 records still carry a mitre_atlas tag today; the other 9 had their tag dropped entirely by
that same audit. The issue itself never recorded which ATLAS release or
commit was fetched — MITRE's real release history
(mitre-atlas/atlas-data releases) shows v2026.07 published 2026-08-07,
two days before the audit merged, making it the very likely candidate,
but "very likely" isn't "recorded," so this goes in as pin_status: "unknown" + read_date: "2026-08-09" once the schema gap above is fixed,
not as an asserted version.
One caveat worth stating plainly: #127's own text says it isn't
exhaustive — it only independently re-checked the four named IDs on each
record, not every mitre_atlas ID that record might also carry (e.g. T0010, T0011, T0052 where already present). framework_sources
records the most recent real verification event touching that field, not
a claim that every individual ID on the record was independently
re-derived that day.
Deferred, explicitly, not silently skipped:
mitre_atlas, 10 records tagged outside the mitre_atlas audit: T0043/T0048/T0051/T0054 mismatched on ~40 of 49 citing records #127 audit
(00024, 00047, 00049, 00051, 00053, 00055, 00066, 00069, 00071, 00076) —
each has only its own record-creation commit as provenance, no dedicated
verification event. Needs individual research, not a bulk date.
nist_ai_rmf, all 56 tagged records — no dedicated audit or
verification event was found anywhere in git history. Every tag's real
provenance is "assigned at record authoring time," which the current
schema (correctly) doesn't treat as a real pin.
owasp_mcp, all 80 records — the field this whole effort traces back
to (OWASP/www-project-mcp-top-10#52's live numbering divergence) has
no corpus-wide audit trail at all, only two doc-only commits (feat: add OWASP MCP Top 10 mapping #15/feat: add OWASP MCP Top 10 mapping #16)
unrelated to per-record tagging. Given this is the actual live case
motivating framework_sources in the first place, it deserves a
dedicated, careful pass verifying each record's owasp_mcp reading
against a specific, real MCP Top 10 commit — not a rushed bulk fill in
the same pass as the other three fields.
Plan
Schema PR: add pin_status: "unknown", update scripts/check_framework_sources.py's has_real_source() to accept it
(same shape as unpinnable: needs read_date, no version/commit
required), update its test.
Backfill PR: 69 owasp_asi entries + 40 mitre_atlas entries, using
only the real dates/versions above. Nothing else touched this pass.
owasp_mcp (80 records) and the remaining mitre_atlas/nist_ai_rmf
gaps stay open, tracked here, for a follow-up pass.
Follow-up to #245/#247 (the schema addition). Doing the backfill surfaced a
real design gap worth fixing before the backfill itself can be honest, plus
the actual archaeology for what's recoverable.
The gap
The schema as merged only allows
pin_status: "unpinnable"— declaringthat a referenced framework "has no version, tag, or commit to pin
against." There is no way to express the very different, and far more
common, fact: the framework does have addressable versions, but which one
a given record's tag was originally checked against predates any tracking
of this at all.
Forcing
unpinnableonto the bulk of the corpus to make the backfilltractable would misrepresent frameworks that plainly do version — MITRE
ATLAS ships monthly releases (
v2026.06,v2026.07, ...), the OWASP Top10 for Agentic Applications prints "Version 2026" on its own cover page,
NIST AI RMF is a dated publication. None of them are actually unpinnable
in the sense the field currently only allows saying.
Proposed fix: add
"unknown"as a secondpin_statusenum value,meaning "the referenced framework has real versions; this record's
provenance predates provenance tracking and the specific version read is
not recoverable." Requires only
read_date(same asunpinnable), nounpinnable_reason(the reason is structural and uniform — "predatestracking" — not per-record). Additive: broadens what a valid record can
express, breaks nothing that currently validates.
Real archaeology, not reconstructed from memory
Every date/version below is from a git commit, a merged PR, an OWASP PDF
fetched fresh this session, or MITRE's own public release history —
checked directly, not recalled.
owasp_asi — 69 of 69 currently-tagged records, real dated event.
The 2026-08-23 audit (#196 + #199, merged same day) verified every
then-existing tagged record's
owasp_asivalue against the primary-sourcePDF (
genai.owasp.org/download/52117). Re-fetched that PDF just now toconfirm its own cover page reads "Version 2026," "December 2025" —
framework_sources.owasp_asi = {"version": "2026", "read_date": "2026-08-23"}is fully real and verifiable for all 69.mitre_atlas — 40 of 50 currently-tagged records, real dated event, no
recoverable exact version. Issue #127 (merged as #164, 2026-08-09)
scored 49 records citing
T0043/T0048/T0051/T0054againstATLAS.yamlfetched live that day. 40 of those 49 records still carry amitre_atlastag today; the other 9 had their tag dropped entirely bythat same audit. The issue itself never recorded which ATLAS release or
commit was fetched — MITRE's real release history
(
mitre-atlas/atlas-datareleases) showsv2026.07published 2026-08-07,two days before the audit merged, making it the very likely candidate,
but "very likely" isn't "recorded," so this goes in as
pin_status: "unknown"+read_date: "2026-08-09"once the schema gap above is fixed,not as an asserted version.
One caveat worth stating plainly: #127's own text says it isn't
exhaustive — it only independently re-checked the four named IDs on each
record, not every
mitre_atlasID that record might also carry (e.g.T0010,T0011,T0052where already present).framework_sourcesrecords the most recent real verification event touching that field, not
a claim that every individual ID on the record was independently
re-derived that day.
Deferred, explicitly, not silently skipped:
(00024, 00047, 00049, 00051, 00053, 00055, 00066, 00069, 00071, 00076) —
each has only its own record-creation commit as provenance, no dedicated
verification event. Needs individual research, not a bulk date.
verification event was found anywhere in git history. Every tag's real
provenance is "assigned at record authoring time," which the current
schema (correctly) doesn't treat as a real pin.
to (
OWASP/www-project-mcp-top-10#52's live numbering divergence) hasno corpus-wide audit trail at all, only two doc-only commits (feat: add OWASP MCP Top 10 mapping #15/feat: add OWASP MCP Top 10 mapping #16)
unrelated to per-record tagging. Given this is the actual live case
motivating
framework_sourcesin the first place, it deserves adedicated, careful pass verifying each record's
owasp_mcpreadingagainst a specific, real MCP Top 10 commit — not a rushed bulk fill in
the same pass as the other three fields.
Plan
pin_status: "unknown", updatescripts/check_framework_sources.py'shas_real_source()to accept it(same shape as
unpinnable: needsread_date, no version/commitrequired), update its test.
owasp_asientries + 40mitre_atlasentries, usingonly the real dates/versions above. Nothing else touched this pass.
owasp_mcp(80 records) and the remainingmitre_atlas/nist_ai_rmfgaps stay open, tracked here, for a follow-up pass.