Skip to content

framework_sources backfill, and a real gap found: pin_status has no 'unknown' state #255

Description

@chaksaray

Follow-up to #245/#247 (the schema addition). Doing the backfill surfaced a
real design gap worth fixing before the backfill itself can be honest, plus
the actual archaeology for what's recoverable.

The gap

The schema as merged only allows pin_status: "unpinnable" — declaring
that a referenced framework "has no version, tag, or commit to pin
against." There is no way to express the very different, and far more
common, fact: the framework does have addressable versions, but which one
a given record's tag was originally checked against predates any tracking
of this at all.

Forcing unpinnable onto the bulk of the corpus to make the backfill
tractable would misrepresent frameworks that plainly do version — MITRE
ATLAS ships monthly releases (v2026.06, v2026.07, ...), the OWASP Top
10 for Agentic Applications prints "Version 2026" on its own cover page,
NIST AI RMF is a dated publication. None of them are actually unpinnable
in the sense the field currently only allows saying.

Proposed fix: add "unknown" as a second pin_status enum value,
meaning "the referenced framework has real versions; this record's
provenance predates provenance tracking and the specific version read is
not recoverable." Requires only read_date (same as unpinnable), no
unpinnable_reason (the reason is structural and uniform — "predates
tracking" — not per-record). Additive: broadens what a valid record can
express, breaks nothing that currently validates.

Real archaeology, not reconstructed from memory

Every date/version below is from a git commit, a merged PR, an OWASP PDF
fetched fresh this session, or MITRE's own public release history —
checked directly, not recalled.

owasp_asi — 69 of 69 currently-tagged records, real dated event.
The 2026-08-23 audit (#196 + #199, merged same day) verified every
then-existing tagged record's owasp_asi value against the primary-source
PDF (genai.owasp.org/download/52117). Re-fetched that PDF just now to
confirm its own cover page reads "Version 2026," "December 2025" —
framework_sources.owasp_asi = {"version": "2026", "read_date": "2026-08-23"} is fully real and verifiable for all 69.

mitre_atlas — 40 of 50 currently-tagged records, real dated event, no
recoverable exact version.
Issue #127 (merged as #164, 2026-08-09)
scored 49 records citing T0043/T0048/T0051/T0054 against
ATLAS.yaml fetched live that day. 40 of those 49 records still carry a
mitre_atlas tag today; the other 9 had their tag dropped entirely by
that same audit. The issue itself never recorded which ATLAS release or
commit was fetched — MITRE's real release history
(mitre-atlas/atlas-data releases) shows v2026.07 published 2026-08-07,
two days before the audit merged, making it the very likely candidate,
but "very likely" isn't "recorded," so this goes in as pin_status: "unknown" + read_date: "2026-08-09" once the schema gap above is fixed,
not as an asserted version.

One caveat worth stating plainly: #127's own text says it isn't
exhaustive — it only independently re-checked the four named IDs on each
record, not every mitre_atlas ID that record might also carry (e.g.
T0010, T0011, T0052 where already present). framework_sources
records the most recent real verification event touching that field, not
a claim that every individual ID on the record was independently
re-derived that day.

Deferred, explicitly, not silently skipped:

  • mitre_atlas, 10 records tagged outside the mitre_atlas audit: T0043/T0048/T0051/T0054 mismatched on ~40 of 49 citing records #127 audit
    (00024, 00047, 00049, 00051, 00053, 00055, 00066, 00069, 00071, 00076) —
    each has only its own record-creation commit as provenance, no dedicated
    verification event. Needs individual research, not a bulk date.
  • nist_ai_rmf, all 56 tagged records — no dedicated audit or
    verification event was found anywhere in git history. Every tag's real
    provenance is "assigned at record authoring time," which the current
    schema (correctly) doesn't treat as a real pin.
  • owasp_mcp, all 80 records — the field this whole effort traces back
    to (OWASP/www-project-mcp-top-10#52's live numbering divergence) has
    no corpus-wide audit trail at all, only two doc-only commits (feat: add OWASP MCP Top 10 mapping #15/feat: add OWASP MCP Top 10 mapping #16)
    unrelated to per-record tagging. Given this is the actual live case
    motivating framework_sources in the first place, it deserves a
    dedicated, careful pass verifying each record's owasp_mcp reading
    against a specific, real MCP Top 10 commit — not a rushed bulk fill in
    the same pass as the other three fields.

Plan

  1. Schema PR: add pin_status: "unknown", update
    scripts/check_framework_sources.py's has_real_source() to accept it
    (same shape as unpinnable: needs read_date, no version/commit
    required), update its test.
  2. Backfill PR: 69 owasp_asi entries + 40 mitre_atlas entries, using
    only the real dates/versions above. Nothing else touched this pass.
  3. owasp_mcp (80 records) and the remaining mitre_atlas/nist_ai_rmf
    gaps stay open, tracked here, for a follow-up pass.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions