fix(headless): mount Maka's build outputs into the task container, not the repo root - #2298
Merged
Merged
Conversation
…t the repo root The Maka arm was exempt from the declared-mount rule on the theory that it runs our own runtime. The model does not know it is ours. In the #2245 two-arm run it grepped the mount and read `docs/eval/terminal-bench-2.1-*.csv` — this benchmark's own per-task pass/fail table from earlier runs — and, on another task, the verifier source at `packages/headless/src/terminal-bench-adapter.ts`. The container executes `dist`, so it now receives dependencies, workspace manifests, build outputs and the one harbor entrypoint the adapter runs. Sources, docs, repo history and the harness manifest are no longer reachable from inside it. Workspace-private `node_modules` are declared but dropped when absent: which ones exist depends on what npm hoisted, and Docker materialises a missing bind source as a new empty directory inside the repo.
The suite checked that an absent workspace dependency tree is dropped. The incident was the opposite: `packages/runtime/node_modules` exists — npm could not hoist `@slack/socket-mode` past a version conflict — and omitting it resolved on the host, then failed in the container on the first import that needed it. Only a live container run caught that. Ask the repo instead.
…obes `install()` tests for both `run-cell.mjs` and `run-host-cell.mjs` before either can run, in every cell-mode branch. Only the first was declared, so the narrowed mount aborted the trial at install rather than degrading it — a regression against the repo-root mount this PR replaces, and one the container smoke check missed because it exercised loading, not installing. The competitor arms already have the check that would have caught it: read the adapter for what it names at a container path and assert the mount declares it. Maka had no equivalent, which is why the omission shipped. It does now — `maka_agent.py` joins path segments onto the mount root instead of writing literals, so the test matches the join. Also asserts the sources-and-records invariant against the mounts actually produced rather than the declared list: a declaration naming no forbidden path still leaks every one of them if the builder mounts the root anyway.
Astro-Han
added a commit
that referenced
this pull request
Aug 6, 2026
…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.
The list lived twice — once implied by the mount declaration, once spelled in the test that cross-checks it against the root manifest. Two copies of an exclusion drift into a workspace nobody mounts.
Astro-Han
added a commit
that referenced
this pull request
Aug 6, 2026
…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.
Astro-Han
marked this pull request as ready for review
August 6, 2026 06:02
Astro-Han
added a commit
that referenced
this pull request
Aug 6, 2026
…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.
Astro-Han
added a commit
that referenced
this pull request
Aug 6, 2026
…aded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label.
Astro-Han
added a commit
that referenced
this pull request
Aug 6, 2026
…aded arm mounts (#2303) * refactor(headless): move the benchmark identity out of the build a graded arm mounts An arm executes this package's dist, so a benchmark's revision, upstream URL and task list compiled into a shipped module are readable from inside the container being scored — which is how the #1970 run turned a task into a lookup at the pinned revision. Narrowing the mount (#2298) closed docs/eval and src; the identity itself still shipped in harness-ab-manifest.js, and the path-level mount checks cannot see it because they inspect declared paths, never what the mounted files contain. The data moves to harbor/benchmark-identity.json, which nothing mounts, and the assertions in src become the comparison without the answer. A content test over the shipped dist now holds the invariant directly; it caught a Terminal-Bench task id hardcoded as a smoke default that the manifest was already supplying. Collapses two identical task-source branches: with each profile carrying its own frozen fingerprint, the full-tree profiles differ only in their label. * refactor(headless): hold the identity boundary where it actually is Review of the first cut found the comments claiming more than the test held. The content check read only the top level of dist while the whole directory is mounted, and described the compiled tests as excluded when they are not. It now scans dist recursively and states two boundaries, because they are two different things: the retrieval keys — revision, upstream URL, tree fingerprint — must be absent from every mounted file, while a single task id is not a retrieval key and is only held out of the shipped modules, where a complete list could accumulate. No mount list may declare the identity file, the Maka one included, and the DeepSWE cache path derives its revision instead of spelling a second copy. * test(headless): check every mounted file, and the fingerprint the list forgot The scan filtered to `.js`, so a source map — whose `sourcesContent` carries the whole of `src` — would have walked straight past it. And the retrieval keys named every task-tree fingerprint but one. * refactor(headless): drop a benchmark task id from a retry comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Maka arm received the repo root at
/opt/maka-agent, exempt from the declared-mount rule every competitor arm follows. The exemption's premise was that Maka runs our own runtime — but the model does not know it is ours, and it reads what is reachable.In the #2245 two-arm run it did exactly that. On
extract-elfit rangrep -rl "extract-elf|extract memory values" /opt/maka-agentand readdocs/eval/terminal-bench-2.1-*.csv— this benchmark's own per-task pass/fail table from earlier runs. Ondb-wal-recoveryit readpackages/headless/src/terminal-bench-adapter.ts, the verifier source. Both cells passed. Neither file is one the runtime executes; both were reachable only because the mount was the root.The container runs
dist, so the arm now gets dependencies, workspace manifests, each workspace's build output, and the single harbor entrypoint the adapter invokes. Sources,docs/, repo history and the harness manifest source are gone from inside it.Refs #2245
Verification
npm run -w @maka/headless test— 1425 pass, 0 fail. Three existing tests asserted the root mount and now assert the narrowed shape.npm run lint,npm run format— cleanThe first attempt failed with
Cannot find package '@slack/socket-mode': npm keeps a privatenode_modulesin any workspace whose version pin it could not hoist, and the list only had the root one. That is exactly the class of failure the contract tests cannot see. With those added, the CLI starts andharbor --helpresolves.Reachability from inside the same container, after the change:
docs/(per-task results)packages/headless/src/(verifier source).git/maka-eval/packages/headless/harbor/run-harness-ab.mjspackages/headless/dist/harness-ab-manifest.jsKnown gap — now closed by #2303
The last row is not fixed here.
harness-ab-manifest.tscompiles TERMINAL_BENCH_2_1_REVISION, the upstream repository URL, the task-tree fingerprint and all 89 task ids intopackages/headless/dist, which the container needs as a directory. Fixing it means moving benchmark identity out of a compiled runtime package, which is a data-layout change rather than a mount change; #2303 does that, and adds the content-level check this PR's path-level assertions structurally cannot make.Review round 2
The suite checked that an absent workspace dependency tree is dropped. The incident was the opposite direction:
packages/runtime/node_modulesexists — npm could not hoist@slack/socket-modepast a version conflict — and omitting it resolved on the host, then failed inside the container on the first import that needed it. Only a live container run caught that.declares every workspace dependency tree npm could not hoist awaynow asks the repo directly, and fails on the pre-fix list. The renderer-only exclusion and the workspace list are also derived from the root manifest in one place instead of two.