Skip to content

api: add --dump option to round-trip harness to write normalized expected/actual xml for failures #210

Description

@webern

Sub-task of #208. See the parent tracking issue for the full plan and context. Builds on the output of #209 (the grown baseline).

Goal

Add a --dump <dir> flag to discovery mode in CorpusRoundtripMain.cpp that writes the fully-normalized expected and actual XML documents for every non-PASS file into a developer-local directory. This makes offline diff analysis possible without touching any checked-in file.

Background

runRoundtrip() in src/private/mxtest/api/CorpusRoundtripMain.cpp currently performs the full pipeline (load → api round-trip → normalize both sides → apply fixup sidecar → compare) and returns only a RoundtripResult (status + one-line detail string). The normalized documents are never surfaced to the caller. Phase 1 needs those documents for the failing files so that Phase 2 (root-cause classification) can diff them without re-running the harness and re-parsing output by hand.

What to implement

1. New dumpDocuments() helper

Add a free function (inside the anonymous namespace in CorpusRoundtripMain.cpp) that takes an absolute path, a dump directory, and a RoundtripResult::Status, re-runs just enough of the pipeline to produce the normalized documents, and serializes them to disk:

void dumpDocuments(const std::string& absolutePath,
                   const std::string& dumpDir,
                   RoundtripResult::Status status);

Do not refactor runRoundtrip(). It has one well-scoped responsibility (run the trip and report pass/fail). Returning intermediate documents from it would widen its interface and leak pipeline internals into the regression-mode path that never needs them. The discovery loop already calls runRoundtrip() for status; when --dump is active it calls dumpDocuments() immediately after for the same file. The duplication of pipeline work (reloading and re-normalizing) is acceptable: --dump is a developer tool, not a hot path.

dumpDocuments() replicates the normalization sequence from runRoundtrip() in the same order:

  1. Load the original input with pugixml → expectedDoc.
  2. Run normalizeForComparison(expectedDoc).
  3. Construct Fixer(absolutePath) and call fixer.applyToExpected(expectedDoc).
  4. Run the api pipeline (same as runRoundtrip()): createFromFile → getData → createFromScore → writeToStream, parse the output string → actualDoc.
  5. Call stripMxAttribution(actualDoc.document_element()).
  6. Run normalizeForComparison(actualDoc).

Then serialize to <dumpDir>/<sanitized-relpath>.expected.xml and <dumpDir>/<sanitized-relpath>.actual.xml (see filename convention below).

For LOADFAIL / GETDATAFAIL / CREATEFAIL statuses the api pipeline did not produce an actual document. In those cases write the expected file only; do not write an actual file and do not fail — print a short note to stderr ("dump: no actual for <relpath> (<STATUS>)").

2. Filename convention

Convert the corpus-relative path to a flat filename by replacing every / (and \ on Windows) with __, then appending .expected.xml or .actual.xml. Example:

lysuite/Saltarello.xml  →  lysuite__Saltarello.xml.expected.xml
                            lysuite__Saltarello.xml.actual.xml

The corpus-relative path is already computed in the discovery loop (fs::relative(absPath, dataRoot).generic_string()). Pass it alongside absolutePath to dumpDocuments().

3. --dump <dir> flag in discovery mode

Extend the main() argument parsing so that discovery mode accepts an optional --dump <dir> argument:

discovery <dataRoot> [--dump <dir>]

Both orderings (--dump before or after <dataRoot>) are acceptable; pick whichever is simplest to parse. The directory is created if it does not exist (std::filesystem::create_directories). If creation fails, print to stderr and exit non-zero.

In the discovery loop, after calling runRoundtrip(), check whether the result is non-PASS and a dump dir is set; if so, call dumpDocuments():

if (dumpDir && r.status != RoundtripResult::Status::pass
             && r.status != RoundtripResult::Status::skip)
{
    dumpDocuments(absPath, relPath, *dumpDir);
}

SKIP files: do not dump. SKIP means the harness intentionally bypassed the file (version-gating or similar) — there is no meaningful expected/actual pair.

PASS files: do not dump. They are already captured in the baseline.

4. Makefile target

Add a dump-api-roundtrip target that builds and runs discovery with --dump pointed at $(BUILD_ROOT)/api/roundtrip-dump/:

# Dump normalized expected/actual XML for every failing api round-trip.
# Output goes to build/api/roundtrip-dump/ (build dir, already gitignored).
dump-api-roundtrip: dev
	mkdir -p $(BUILD_ROOT)/api/roundtrip-dump
	$(BUILD_ROOT)/api/mxtest-api-roundtrip discovery $(CURDIR)/data \
		--dump $(CURDIR)/$(BUILD_ROOT)/api/roundtrip-dump

Place it immediately after the discover-api-roundtrip target (around line 179 of the current Makefile).

What gets checked in

  • src/private/mxtest/api/CorpusRoundtripMain.cpp — the only source file changed.
  • Makefile — the new dump-api-roundtrip target.

Nothing else. The dump output files are ephemeral developer artifacts. They live in build/api/roundtrip-dump/, which is inside build/ — already gitignored. They must never appear in git status after a run.

Dump output location

$(BUILD_ROOT)/api/roundtrip-dump/ — equivalently build/api/roundtrip-dump/ from the repo root. The build/ tree is already excluded by .gitignore; no new ignore entry is needed. A developer can also pass any other path to --dump directly.

Definition of done

  • make dump-api-roundtrip runs without error and populates build/api/roundtrip-dump/ with .expected.xml and .actual.xml pairs for each FAIL file.
  • LOADFAIL / GETDATAFAIL / CREATEFAIL files produce a .expected.xml only; no .actual.xml is written and no error is returned for those.
  • PASS and SKIP files produce no output in the dump directory.
  • The normalized expected document written to disk matches what compareElements() would have seen on the expected side: normalizeForComparison applied, then Fixer::applyToExpected applied.
  • The normalized actual document written to disk matches what compareElements() would have seen on the actual side: stripMxAttribution applied, then normalizeForComparison applied.
  • make test-api-roundtrip passes (the regression gate is unaffected).
  • make check passes.
  • git status is clean after make dump-api-roundtrip (no dump files, no accidental source changes).
  • Commit references api: add --dump option to round-trip harness to write normalized expected/actual xml for failures #210; this issue is checked off in api: expand corpus coverage of the api round-trip pass-list #208.

Metadata

Metadata

Assignees

No one assigned

    Labels

    aiIssues opened by, or through, a coding agent.apiAffects the mx::api layernon-breakingfixes or implementation that do not require breaking changestesting

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions