api: classify corpus round-trip failures by root cause (drop-only, reorder, enum bug, missing feature)

[webern]

Sub-task of #208. See the parent tracking issue for the full plan and context. Builds on #209 (the grown baseline) and #210 (the --dump flag that produces normalized expected/actual XML pairs).

⚠️ Design revision (2026-06-20) — read this first

The original algorithm below (step 3b: "walk both trees in parallel until the first mismatch") has a structural defect: it is a positional, paired, depth-first walk — the same shape as compareElements in corert/Compare.cpp — and it cannot survive a deletion. mx::api drops subtrees and reorders children by design, so the dominant signal is a missing element. The moment actual is missing one child, the two child-lists shift out of alignment and every later sibling is compared against the wrong partner; only the first divergence is trustworthy. A single drop poisons the rest of the file. This is the classic "diff by index" cascade — the fix is alignment (LCS/Myers) or, since we don't need positions, an order-free multiset comparison.

Full rationale, algorithm walkthrough, library survey, and cited research are in docs/ai/design/api-roundtrip-classifier.md. Summary of the revised design:

Use a multiset-first, layered diff (stdlib only: xml.etree.ElementTree + collections.Counter + difflib). No lxml/xmldiff/zss/apted.

• Layer 1 — multiset tag diff (primary). missing = Counter(expected tags) - Counter(actual tags) (Counter subtraction keeps only positive counts) enumerates every dropped element class in O(n), fully reorder-invariant. added = Counter(actual) - Counter(expected) catches spurious tags. This replaces the parallel walk in step 3b.
• Layer 2 — path-qualified multiset. Re-key by parent/tag (or full path) for blocking_features attribution; headline metric stays on bare tags.
• Layer 3 — cross-reference api.features.xml. Category B (drop-only) becomes provable across the whole file: it holds iff every tag in missing has support="none". Any missing tag with support="full"/"partial" is a real bug/partial-drop, surfaced instead of hidden behind the first divergence.
• Layer 4 — difflib.SequenceMatcher per parent (category C only). Reorder-only = parent's child multiset equal but child sequence differs (a pure permutation).
• Layer 5 — edit-distance scalar = sum(missing.values()) as a secondary "distance to passing" tie-breaker only. Not the ranking key; no tree-edit-distance library.

The key question — "how many distinct element classes are missing" — is answered directly: distinct_missing_count = len(missing). Low-hanging fruit = files where distinct_missing_count == 1 (missing exactly one class, once or many times); group those by the single missing tag to get the prioritized worklist. Per the regression-triage literature (e.g. BuildSheriff, ICSE'22), rank features by files unblocked, not raw occurrence count.

New per-file JSON fields (added to the schema below; existing fields retained, first_divergence_* kept for continuity but no longer relied on for completeness):

"missing_elements": ["credit", "defaults"],        // sorted distinct dropped tags
"missing_element_counts": {"credit": 3, "defaults": 1},
"distinct_missing_count": 2,                         // headline ranking metric
"added_elements": [],                               // spurious tags in actual (bug signal)
"is_single_blocker": false,                         // distinct_missing_count == 1
"total_missing_instances": 4                        // sum of counts; severity scalar

The rest of the spec (invocation, inputs, categories C–F, Makefile target, what-gets-checked-in, DoD) is unchanged. The original text is preserved below.

---

Goal

Build a failure classifier in audit/classify.py that reads the dump directory produced by Phase 1 (make dump-api-roundtrip), cross-references data/api.features.xml and the per-file *.features.xml sidecars, and assigns each non-passing corpus file one primary root-cause category plus any secondary categories. Output is a machine-readable JSON file (consumed by Phase 3 ranking) with a human-readable summary printed to stdout.

Background

make dump-api-roundtrip (Phase 1) writes two files per FAIL:

build/api/roundtrip-dump/lysuite__Saltarello.xml.expected.xml
build/api/roundtrip-dump/lysuite__Saltarello.xml.actual.xml

LOADFAIL / GETDATAFAIL / CREATEFAIL produce only an .expected.xml file (no .actual.xml). PASS and SKIP files produce nothing.

The corpus relative path is recovered by splitting on __ and stripping the trailing .expected.xml / .actual.xml suffix. Given lysuite__Saltarello.xml.expected.xml, the relative path is lysuite/Saltarello.xml.

The harness's one-line FAIL detail string (from compareElements in corert/Compare.cpp) has four shapes:

Mismatch type	Detail prefix
Element name mismatch	element name mismatch at /path/to/el: expected 'X', actual 'Y'
Text mismatch	text mismatch at /path/to/el: expected 'X', actual 'Y'
Attribute mismatch	attribute mismatch at /path/to/el: expected 'name=value', actual 'name=value'
Child count mismatch	child count mismatch at /path/to/el
Attribute count mismatch	attribute count mismatch at /path/to/el

The path embedded in the detail string (e.g. /score-partwise/part/measure/note/accidental) names the deepest element that diverged. The classifier uses the leaf element name from this path as the first_divergence_element.

Note: the detail string captures only the first mismatch found by depth-first traversal. This is exactly why the classifier must not re-derive divergences the same way — see the design revision at the top. The full XML diff (from the dump files), analyzed as a multiset, is needed to determine the complete set of missing element classes.

What to implement

New file: audit/classify.py

A new module that fits the existing python3 -m audit pattern. Expose it as the classify subcommand by adding it to audit/__main__.py.

Invocation

python3 -m audit classify <dump_dir> [--data <data>] [--out <out>]

• <dump_dir> — required; the directory containing the dump files from Phase 1 (e.g. build/api/roundtrip-dump/). The classifier infers file status from which dump files are present.
• --data — optional; defaults to data/ relative to the repo root (resolved the same way other audit modules resolve DATA_ROOT).
• --out — optional; defaults to build/api/classified.json. The directory is created if it does not exist.

Inputs

The classifier reads:

1. The dump directory — pairs of *.expected.xml / *.actual.xml files. Filename encodes the corpus-relative path (replace __ with / and strip the status suffix). Presence of .actual.xml implies FAIL status; absence (with .expected.xml present) implies LOADFAIL / GETDATAFAIL / CREATEFAIL.
2. data/api.features.xml — the mx::api support index. For each <feature>, reads support attribute (full / partial / none), per-attribute <attribute>, and <missing> enum members within <enum> blocks. Load once at startup.
3. Per-file *.features.xml sidecars (e.g. data/lysuite/Saltarello.features.xml) — the element/attribute surface of the original file, produced by python3 -m audit files. Used to determine which features the file exercises, to cross-reference support levels. Read via the existing featuresfile.read() function. If a sidecar does not exist, log a warning and skip cross-referencing for that file (don't abort).

The classifier does not re-run the C++ harness. It works entirely from the dump files and the existing data artifacts.

Classification categories

Assign each file exactly one primary category and zero or more secondary categories:

ID	Name	Condition
A	already-passing	.expected.xml present but neither .expected.xml nor .actual.xml suggests a failure — indicates a file that PASS'd after Phase 0 was run but somehow appears in the dump. Handle gracefully; do not error.
B	drop-only	FAIL; the actual is a structural subset of expected, and every element class in the multiset difference missing = Counter(expected) - Counter(actual) has support="none" in api.features.xml. No element with support="full" or support="partial" appears in missing. (Revised: provable across the whole file via the multiset, not just the first divergence.)
C	reorder-only	FAIL; the sets of element names at a given parent match between expected and actual but their order differs. Detected when a parent's child multiset is equal between expected and actual but the child-tag sequence differs (a pure permutation, confirmed via difflib.SequenceMatcher opcodes).
D	enum-bug	FAIL; detail is text mismatch or attribute mismatch; the diverging element/attribute is mapped to a feature with support="partial" that has <missing> enum members, and the expected value matches one of those missing members (the api silently maps it to the first/wrong variant).
E	missing-attribute	FAIL; detail is attribute mismatch or attribute count mismatch; the diverging element is mapped to a feature with support="partial" whose <attribute> entry has support="partial" for the missing attribute. The surrounding feature is modeled but this one attribute is dropped.
F	pipeline-error	Status is LOADFAIL, GETDATAFAIL, or CREATEFAIL (only .expected.xml present; no .actual.xml). Real bugs; triage separately from feature gaps.

Classification algorithm (per file):

1. Parse the dump filename to recover corpus-relative path and status (F if no .actual.xml, else proceed to diff).
2. For F (pipeline error): assign primary=F, no secondary; record the status code (LOADFAIL / GETDATAFAIL / CREATEFAIL) as a field.
3. For FAIL files (.actual.xml present):
   a. Parse both XML files with xml.etree.ElementTree.
   b. (REVISED — see design revision at top.) Build the tag multisets for both trees and compute missing = Counter(expected) - Counter(actual) and added = Counter(actual) - Counter(expected). This enumerates all dropped/spurious element classes, not just the first. Record missing_elements, missing_element_counts, distinct_missing_count, added_elements, is_single_blocker, total_missing_instances. Separately, retain a first_divergence_element/first_divergence_path/mismatch_type from a single positional pass (or the harness detail string) only for continuity — do not rely on it for completeness.
   c. Look up each element class in missing (and the first-divergence element) in api.features.xml → get support.
   d. Apply category rules B → C → D → E in priority order (first match wins for primary). Record all that apply as secondary.
   e. If none of B–E match (e.g., support="full" but content differs — a genuine correctness bug), assign primary=unknown and log to stderr.
4. Build the blocking-feature list: the set of api.features.xml feature names that, if fully supported, would eliminate this file's primary blocker — derived from missing cross-referenced to features, prioritizing the single-blocker case.

Output: JSON schema

Write a single JSON object:

{
  "dump_dir": "/abs/path/to/build/api/roundtrip-dump",
  "data_root": "/abs/path/to/data",
  "generated": "ISO-8601 timestamp",
  "summary": {
    "total": 123,
    "by_category": {
      "B": 45,
      "C": 12,
      "D": 8,
      "E": 31,
      "F": 7,
      "unknown": 20
    }
  },
  "files": [
    {
      "file": "lysuite/Saltarello.xml",
      "status": "FAIL",
      "primary_category": "B",
      "secondary_categories": [],
      "first_divergence_element": "part-group",
      "first_divergence_path": "/score-partwise/part-list/part-group",
      "mismatch_type": "element-name",
      "missing_elements": ["part-group"],
      "missing_element_counts": {"part-group": 1},
      "distinct_missing_count": 1,
      "added_elements": [],
      "is_single_blocker": true,
      "total_missing_instances": 1,
      "blocking_features": ["part-group"],
      "pipeline_error_kind": null
    },
    {
      "file": "foundsuite/An Chloe.xml",
      "status": "LOADFAIL",
      "primary_category": "F",
      "secondary_categories": [],
      "first_divergence_element": null,
      "first_divergence_path": null,
      "mismatch_type": null,
      "missing_elements": [],
      "missing_element_counts": {},
      "distinct_missing_count": 0,
      "added_elements": [],
      "is_single_blocker": false,
      "total_missing_instances": 0,
      "blocking_features": [],
      "pipeline_error_kind": "LOADFAIL"
    }
  ]
}

All string fields are present on every entry (use null when not applicable). status is one of FAIL, LOADFAIL, GETDATAFAIL, CREATEFAIL. primary_category is one of B, C, D, E, F, unknown. mismatch_type is one of element-name, text, attribute, attribute-count, child-count, null.

Stdout summary

After writing the JSON, print a human-readable summary to stdout:

Classified 123 files from build/api/roundtrip-dump/

  B  drop-only divergence        45
  C  reorder-only divergence     12
  D  enum bug                     8
  E  missing attribute/element   31
  F  pipeline error               7
  ?  unknown                     20

Top blocking features (ranked by files unblocked; B+D+E):
  credit              38 files   (12 single-blocker)
  defaults            27 files   ( 9 single-blocker)
  part-group          22 files   ( 7 single-blocker)
  ...

Output: build/api/classified.json

"Top blocking features" lists the blocking_features entries most frequently appearing across B, D, and E files (the actionable ones), sorted descending by files unblocked, with the single-blocker count shown as the low-hanging-fruit tie-breaker, capped at 15.

Changes to audit/__main__.py

Add classify as a new subcommand alongside files, corpus, and all. Import classify from the new module. Wire up <dump_dir>, --data, and --out arguments as described above.

Makefile target

Add classify-api-roundtrip immediately after dump-api-roundtrip:

# Classify api round-trip failures by root cause.
# Reads the dump produced by dump-api-roundtrip; writes build/api/classified.json.
# Requires: make dump-api-roundtrip first (or pass DUMP_DIR=path explicitly).
DUMP_DIR ?= $(BUILD_ROOT)/api/roundtrip-dump
classify-api-roundtrip:
	python3 -m audit classify $(DUMP_DIR) \
		--data $(CURDIR)/data \
		--out $(BUILD_ROOT)/api/classified.json

The two targets are kept separate rather than chained. The dump step is slow (it runs every corpus file through the C++ api pipeline). The classify step is fast (pure Python, reads existing files). Separating them lets a developer re-run classification with different logic without re-dumping. A developer who wants both in sequence runs make dump-api-roundtrip && make classify-api-roundtrip.

What gets checked in

Source only:

• audit/classify.py — the classifier module.
• audit/__main__.py — updated to wire in the classify subcommand.
• Makefile — the classify-api-roundtrip target.
• docs/ai/design/api-roundtrip-classifier.md — the diff-design rationale (added by the design revision).

No JSON output checked in. The JSON output lives in build/api/classified.json, which is inside build/ — already gitignored. It must not appear in git status after a run. No new .gitignore entry is needed.

Why the output is not checked in

The JSON is an intermediate artifact consumed by Phase 3 ranking. It is derived entirely from (a) the dump directory (ephemeral, build-output) and (b) data/api.features.xml (checked in, regenerable). There is no permanent consumer. A future make classify-api-roundtrip re-derives it in seconds. Checking it in would add a stale artifact that diverges from the dump directory the moment any api code changes — the same reasoning that keeps the dump dir itself out of the tree (see Phase 1, #210).

Definition of done

• audit/classify.py implemented; importable as python3 -m audit classify.
• Diff is multiset-based (enumerates all missing element classes), per the design revision; docs/ai/design/api-roundtrip-classifier.md added.
• All 6 harness status codes handled: PASS (graceful), FAIL, SKIP (graceful), LOADFAIL, GETDATAFAIL, CREATEFAIL → category F.
• All 5 failure categories produced (B, C, D, E, F) plus unknown for unclassified FAIL files.
• JSON output matches the schema above, including the new missing_elements / missing_element_counts / distinct_missing_count / added_elements / is_single_blocker / total_missing_instances fields (all fields present on every entry; null/[]/{} for non-applicable).
• Stdout summary printed: counts per category + top blocking features ranked by files unblocked with single-blocker counts.
• --data defaults correctly to the repo's data/ directory; --out defaults to build/api/classified.json; directory is created if absent.
• Missing per-file sidecar is a warning, not an abort.
• audit/__main__.py updated: classify subcommand wired.
• make classify-api-roundtrip target added to Makefile.
• make test-api-roundtrip passes (no regression in the CI gate).
• make check passes.
• git status is clean after make dump-api-roundtrip && make classify-api-roundtrip (no output files, no accidental source changes).
• Sample output (classified.json contents and the stdout summary) posted in this issue or the linked PR as evidence of a real run.
• Commit references api: classify corpus round-trip failures by root cause (drop-only, reorder, enum bug, missing feature) #211; this issue is checked off in api: expand corpus coverage of the api round-trip pass-list #208.