Skip to content

fix(headless): teach the trajectory exporter the two envelope fields it never learned - #2314

Merged
Astro-Han merged 2 commits into
mainfrom
fix/harness-maka-trajectory-envelope-drift
Aug 6, 2026
Merged

fix(headless): teach the trajectory exporter the two envelope fields it never learned#2314
Astro-Han merged 2 commits into
mainfrom
fix/harness-maka-trajectory-envelope-drift

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

What

Every Maka cell of the #2245 two-arm run exported a one-line summary in place of its trajectory — 89 of 89, all with maka_summary_reason: runtime_event_schema_invalid. Codex exported 89 of 89 complete, 15 to 184 steps each.

That means any analysis built on the normalized ATIF trajectory is blind on our own arm, which is how this surfaced: it is the data source a contamination detector would read.

Refs #2245

Root cause

maka_trajectory.py's _is_runtime_event validates the RuntimeEvent envelope against a hand-copied key whitelist and rejects any event carrying a key it does not list. The runtime's envelope gained origin and modelVisibility; the whitelist never learned them.

Replaying the run's own runtime-events.jsonl, all 89 files fail at the same shape, and always with the same two keys:

89 files, 89 rejected — extra keys: modelVisibility, origin
first: adaptive-rejection-sampler, event 4 of 507
  { ..., "role": "model", "author": "agent",
    "origin": "provider", "modelVisibility": "visible", ... }

Both are legitimate optional envelope fields — RUNTIME_EVENT_SHAPE in packages/core/src/runtime-event.ts lists them, and defineObjectShape<RuntimeEvent>() type-forces that list to cover the interface. Only the Python copy could drift, and it did. In this corpus the values are only ever provider/visible or absent, so nothing semantic was lost — the events were simply refused.

Why the existing guard missed it

Core's decoder and the Python exporter already share one validation corpus (runtime-event-validation-corpus.json), run from both sides. It did not catch this because the corpus is a hand-written list of cases: 52 of them, and four envelope keys — origin, modelVisibility, branch, status — set by none of them. It covered the keys someone thought to write cases for.

So the fix is in two parts:

  • maka_trajectory.py accepts both fields with the value constraints the TypeScript decoder applies (provider/code_mode, visible/hidden).
  • The corpus covers all four previously unexercised keys, accepted and rejected, and a new Core test holds the corpus to runtimeEventEnvelopeKeys(). TypeScript already forces that list to cover the interface, so the chain is now: interface → shape → corpus → Python. A field added without a case fails in Core rather than silently blinding a benchmark run.

Verification

Replaying all 89 cells' real runtime-events.jsonl through the exporter, with the run's own status and runtime refs:

before after
full trajectory 0 78
runtime_event_schema_invalid 89 0
image_artifact_metadata_missing 0 11

Recovered trajectories run 9 to 252 steps (median 26).

The 11 remaining cells carry image content whose artifact metadata resolves against a trajectory-state store that is not present in any of the 89 exported cells, so this corpus cannot say whether those 11 would have exported fully in production — a different reason code and a separate question.

Red/green, both directions:

  • Removing the two keys from the Python whitelist again → the corpus-driven Python test fails on valid-provider-origin.
  • Removing the origin corpus cases → the new Core test fails with no corpus case sets the envelope key origin.

npm run -w @maka/core test — 796 pass. npm run -w @maka/headless test — 1421 pass. Lint and format clean.

…it never learned

Every Maka cell of the #2245 benchmark run exported a one-line summary instead
of a trajectory — 89 of 89, against Codex's 89 of 89 complete. The exporter's
envelope check is a hand-copied key whitelist that never learned `origin` and
`modelVisibility`, so the first event carrying either failed the check and the
whole run degraded. Replaying the run's own runtime events through the fixed
check turns 0 full trajectories into 78; the remaining 11 need an artifact
store the export does not carry, so they cannot be judged from it.

The shared validation corpus exists to stop exactly this, and could not: it
exercised the keys someone thought to write cases for, and four envelope keys
— `origin`, `modelVisibility`, `branch`, `status` — were never among them. The
corpus now covers all four, and a contract test holds it to the envelope shape
that TypeScript already forces to cover the interface. A field added without a
case now fails in Core rather than silently blinding a benchmark run.
@Astro-Han
Astro-Han force-pushed the fix/harness-maka-trajectory-envelope-drift branch from 3627391 to 41fd5c2 Compare August 6, 2026 06:26
@Astro-Han
Astro-Han marked this pull request as ready for review August 6, 2026 06:36
@Astro-Han
Astro-Han merged commit 1e1c552 into main Aug 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant