feat(lab): @maka/lab — headless agent experiment lab (MVP, #31) - #36
Closed
Astro-Han wants to merge 6 commits into
Closed
feat(lab): @maka/lab — headless agent experiment lab (MVP, #31)#36Astro-Han wants to merge 6 commits into
Astro-Han wants to merge 6 commits into
Conversation
#31) First end-to-end slice of the agent experiment lab: run one Config against one Task in an isolated sandbox, capture the trajectory, score it, and return a ResultRecord. De-risks the integration (headless SessionManager run + sandbox + evaluator composing into one real run); matrix/compare/CLI and real-model backends are later additions. - contracts.ts: Task (instruction + fixture workspace + verification command), Config (backend/connection/model), ResultRecord. Minimal; systemPrompt/Execution/SWE-bench-pack ingestion deferred. - sandbox.ts: prepareWorkspace copies the fixture to a throwaway dir so the agent never mutates the source (isolation, not asking). - evaluator.ts: runVerification runs the Task's test command in the throwaway workspace after the agent finishes (config can't grade itself), with output cap + timeout kill. - runner.ts: runExperiment wires a pure-Node SessionManager (injected registerBackends keeps model/credential wiring out of the lab core), drives one turn, captures the InvocationResult trajectory via runtimeInvocationObserver, scores it, returns a ResultRecord. - tests: FakeBackend e2e (pass + fail fixtures, trajectory persisted as runtime-events.jsonl), sandbox isolation, evaluator pass/fail/timeout. Independent of the credential migration (#32/#33): the skeleton runs on FakeBackend and needs no credentials.
Completes the MVP loop on top of the walking skeleton: run a grid of Configs × Tasks, persist canonical results, and compare. - matrix.ts: runMatrix runs the full cross product (sequential; a thrown run becomes a failed cell instead of aborting the grid) with a per-cell onResult callback. - results.ts: ResultRecord JSONL is canonical truth; toComparisonTable derives a git-diffable markdown grid (tasks × configs, ✅/❌/⚠️ , pass-rate footer). ResultRecord gains an optional `error`. - backends.ts: the two concrete backend wirings, kept out of the engine. registerFakeBackend (deterministic). registerAiSdkBackend resolves a Config's slug against spec connections, reads the API key from a named env var (no secrets at rest), and wires a minimal AiSdkBackend (model + builtin tools + execute-mode permission). Telemetry/artifact/synthesis hooks omitted — a benchmark scores via the verification command. - runner.ts: the drain loop now auto-approves permission requests — a headless benchmark has no human to confirm; throwaway-workspace isolation is the safety net. - cli.ts: `maka-lab run <spec.json> [--out <dir>]` and `maka-lab compare <results.jsonl>`; task fixtures resolve relative to the spec. Exposed as the `maka-lab` bin. - tests (15 total): CLI end-to-end smoke (spawns the built bin on a fake spec → results.jsonl + comparison.md → compare), matrix cross-product + failed-cell, JSONL round-trip, table rendering. The real ai-sdk backend is typecheck-verified but not unit-tested (a live model call is non-deterministic and costs money); it needs a live smoke test with a real API key. Everything else is deterministic and green.
README documents the Config × Task model, the CLI (run/compare), the spec shape (incl. a real ai-sdk connection with apiKeyEnv), the two backends, and what's deliberately out of MVP scope. examples/demo.spec.json + examples/demo/marker.txt run green on the fake backend with no API key: maka-lab run examples/demo.spec.json --out /tmp/maka-lab-demo
examples/fix-add — a buggy add() with a failing node:test. A real model run must read the files, fix src.mjs, and turn `node --test` green. Live smoke confirmed end-to-end on DeepSeek (deepseek-chat): completed, passed, exit 0, 124-event trajectory using Edit/Write/Bash; the source fixture stayed buggy (the agent only touched the throwaway copy), proving sandbox isolation + headless permission auto-approve work against a real backend. README points at it as the canonical real-run example.
…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.
The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.
jackwener
pushed a commit
that referenced
this pull request
Jun 21, 2026
…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.
jackwener
pushed a commit
that referenced
this pull request
Jun 21, 2026
The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.
jackwener
pushed a commit
that referenced
this pull request
Jun 21, 2026
…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.
jackwener
pushed a commit
that referenced
this pull request
Jun 21, 2026
The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.
jackwener
pushed a commit
that referenced
this pull request
Jun 21, 2026
…r, table (#36) - runner: stop blanket-approving permission prompts. Allow ordinary tool use, DENY dangerous categories (fs_destructive / git_destructive / privileged / browser) by default — the workspace is a copy, not a jail, so a tool can still escape via absolute paths/network. Opt in with allowDangerousTools (real sandbox only). Corrects the misleading "workspace is the safety net" comment. Also: a run that didn't complete can no longer read as passed. - evaluator: spawn detached + SIGKILL the process group on timeout, so backgrounded grandchildren die too (a plain child.kill leaked them). - sandbox: reject fixture symlinks (fs.cp preserved them verbatim → escape to source/host) and clean up the temp dir if the copy fails (it was leaked before the runner's finally registered). - results: cellKey was separated by a NUL byte, making results.ts a binary file to git — switch to a JSON key. Render a failed run as⚠️ (distinct from ❌ verification-failed) and exclude it from the pass count. Escape `|`/newline in ids so they can't break the table. - cli: reject unknown flags and require a value after --out. - README: honest permission/safety wording. +5 tests (symlink reject, failed-run render, id escaping, unknown flag, missing flag value); 20 total green. Cross-process credential lock and a real container sandbox remain deliberately out of scope.
jackwener
pushed a commit
that referenced
this pull request
Jun 21, 2026
The throwaway workspace stops a run from mutating the source fixture, but it is NOT a security sandbox and a config could rewrite its own test to pass. This addresses the two open review P1s without ripping out the real backend (kept usable for your-own-models-on-trusted-tasks): - Clean-room grading (verification integrity): Task.verification.protectedPaths lists the grading assets; the runner restores them from the pristine fixture AFTER the agent finishes and BEFORE the verification command runs, so a model that rewrote its own test has that edit reverted. Each path is removed first (drops an agent-planted symlink) and rejected if it escapes the workspace. examples/fix-add now protects test.mjs. - Honest docs (host isolation): the README states plainly that a real run executes tool calls on your machine with your privileges — same exposure as running Maka directly — so run only models/tasks you trust. Per-run container isolation (mount workspace only, env allowlist, network policy) is the named next hardening; allowDangerousTools is for inside it. Tests (+4, 24 total green): restoreProtectedPaths unit (reverts protected, keeps the rest, rejects escapes) + a malicious TamperBackend integration proving protectedPaths reverts the cheat (passed=false) while the unguarded task lets the cheat win (passed=true) — the guard is load-bearing. The integration grades via `node check.mjs` (exit-code), not `node --test`, so the verification child doesn't collide with the lab's own test runner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the MVP from #31. Independent of #32 / #33 — the lab runs on the FakeBackend with no credentials, and the real backend reads keys from an env var, not the credential store.
What & why
A headless lab for measuring an agent configuration: run a Config × Task grid in isolated sandboxes, capture each trajectory, score it with the task's own test command, and compare. Vary the model/connection (Config), hold the work (Task), read pass/fail off the table.
What's in this PR
New package
@maka/lab(added to the workspaces list):Task(instruction + fixture workspace + verification command),Config(backend / connection / model),ResultRecord(one canonical row per run). Minimal on purpose; kept in@maka/labfor now, promotable to@maka/coreonce a second consumer exists.SessionManager, captures the trajectory viaruntimeInvocationObserver, and auto-approves permission prompts (a benchmark has no human to confirm; the throwaway workspace is the boundary).Config × Taskcross product; a thrown run becomes a failed cell instead of aborting the grid.fake(deterministic) andai-sdk(real model; key from a named env var, minimal AiSdkBackend = model + builtin tools + execute-mode permission).maka-lab run <spec.json> [--out <dir>]andmaka-lab compare <results.jsonl>.demo.spec.json(fake, no key) andfix-add.spec.json(a real coding task).Live validation
The real
ai-sdkbackend was smoke-tested end-to-end on DeepSeek (deepseek-chat) withexamples/fix-add(a buggyadd()+ a failingnode:test):Edit/Write/BashTesting
15 deterministic tests (FakeBackend), all green: CLI end-to-end smoke (spawns the built bin →
results.jsonl+comparison.md→compare), matrix cross-product + failed-cell, JSONL round-trip, table rendering, sandbox isolation, evaluator pass/fail/timeout, walking-skeleton e2e. Plus the live DeepSeek smoke above.Out of scope (deliberate — pure additions later)
Parallel matrix execution, LLM/rule evaluators (today: command exit code), Docker / network isolation, SWE-bench pack ingestion, a richer report than the markdown grid, and promoting the contracts into
@maka/core.