fix(headless): teach the trajectory exporter the two envelope fields it never learned - #2314
Merged
Merged
Conversation
…it never learned Every Maka cell of the #2245 benchmark run exported a one-line summary instead of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's envelope check is a hand-copied key whitelist that never learned `origin` and `modelVisibility`, so the first event carrying either failed the check and the whole run degraded. Replaying the run's own runtime events through the fixed check turns 0 full trajectories into 78; the remaining 11 need an artifact store the export does not carry, so they cannot be judged from it. The shared validation corpus exists to stop exactly this, and could not: it exercised the keys someone thought to write cases for, and four envelope keys — `origin`, `modelVisibility`, `branch`, `status` — were never among them. The corpus now covers all four, and a contract test holds it to the envelope shape that TypeScript already forces to cover the interface. A field added without a case now fails in Core rather than silently blinding a benchmark run.
Astro-Han
force-pushed
the
fix/harness-maka-trajectory-envelope-drift
branch
from
August 6, 2026 06:26
3627391 to
41fd5c2
Compare
Astro-Han
marked this pull request as ready for review
August 6, 2026 06:36
This was referenced Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with
maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.
Refs #2245
Root cause
maka_trajectory.py's_is_runtime_eventvalidates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gainedoriginandmodelVisibility; the whitelist never learned them.Replaying the run's own
runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:Both are legitimate optional envelope fields —
RUNTIME_EVENT_SHAPEinpackages/core/src/runtime-event.tslists them, anddefineObjectShape<RuntimeEvent>()type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only everprovider/visibleor absent, so nothing semantic was lost — the events were simply refused.Why the existing guard missed it
Core's decoder and the Python exporter already share one validation corpus (
runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys —origin,modelVisibility,branch,status— set by none of them. It covered the keys someone thought to write cases for.So the fix is in two parts:
maka_trajectory.pyaccepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode,visible/hidden).runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.Verification
Replaying all 89 cells' real
runtime-events.jsonlthrough the exporter, with the run's own status and runtime refs:runtime_event_schema_invalidimage_artifact_metadata_missingRecovered trajectories run 9 to 252 steps (median 26).
The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.
Red/green, both directions:
valid-provider-origin.origincorpus cases → the new Core test fails withno corpus case sets the envelope key origin.npm run -w @maka/core test— 796 pass.npm run -w @maka/headless test— 1421 pass. Lint and format clean.