Eval frontier: fair same-session A/B (graff-dev hardlink + grok + OpenCode --dir) - #687
Merged
Conversation
Add opencode / opencode-zen / dsh-grok next to grok-build, plus six self-contained fixtures distilled from shipped CodeGraff bugs (symlink write, oneshot chrome, cache git-root, stall widen/warn, empty catalog). Same-model grok-4.6 and native-default series are labeled separately.
dsh does not read graff/grok OAuth files. Add a machine-local hook that copies the existing SuperGrok seat into XAI_API_KEY and $DSH_HOME/.credentials.yaml, plus the non-secret X-XAI-Token-Auth routing header on the xAI patches so login tokens are accepted.
Owner
Author
dsh clone + local SuperGrok OAuth (sibling)Owned the dsh install/auth slice so the live A/B can actually run dsh. Did not redo the OpenCode / in-house graph. Install (this machine, not vendored)
Auth — same SuperGrok seat as graff/grok, locally only
Include in the frontier run:
Note for the sibling: graff-evals/dsh-local-oauth.md. |
Spine (grok-4.6): graff-dev 3/3 · $0.0481 · 8 calls; grok-build 3/3 · 31.6s wall; OpenCode 2/3 · $0.43. In-house PR suite: graff-dev 6/6, grok-build 5/6, OpenCode 0/6. dsh ran grok-4.5 only (catalog has no 4.6). Do not plot REPL $0 or mixed-model points as grok-4.6 list price.
…t server
OpenCode `run --format json` binds a project root; Popen cwd is not enough,
so file-ops wrote nowhere the check could see. Pass --dir {sandbox} --pure.
When the JSON pipes close the local server often stays up — that is not a
timeout; wait a beat for writes, then reap the process group.
The earlier 2/3 and 0/6 were our harness: writes missed the check cwd. Rerun run-20260830-095006 / 095035. OpenCode now wins wall; graff still wins list$ / RSS. Not Pareto.
Fair same-model A/B: graff-dev must not pay a 127M byte-copy + 38s suite gen on teardown. Same pin/init path as PR 684.
Live 27-run comparison of graff-dev, grok-build, and OpenCode --dir on one SuperGrok seat. Pin is a hardlink (nlink=7, same inode). Old copy-tax / bad --dir tables stay labeled as prior. Not Pareto.
cursor Bot
pushed a commit
that referenced
this pull request
Aug 30, 2026
Tier 1 warned the ratchet sat 33 ahead of 1758 (slack 25) after folding #687. Bump the floor on this cut so a dropped module would still trip the hook. Notes mention the eval-frontier land.
This was referenced Aug 30, 2026
yermakoffivan
pushed a commit
to yermakoffivan/codegraff
that referenced
this pull request
Aug 31, 2026
Double --suite inhouse with distilled verifiers from merged CodeGraff PRs: hardlink pin (justrach#687), hosted x_search splice (justrach#632), first-turn MCP skip (ADR 0035/0029), rlm showcase gate (justrach#633), codedb five-shot menu (justrach#597), peer-channel resume (justrach#584). Same pattern as the first six — buggy starter + SPEC + visible tests + hidden/ check. Not the live repo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fair same-session grok-4.6 A/B is done. 27 live provider runs on one SuperGrok seat after a forced OAuth refresh. Hosted
x_searchstays on. Do not plot dsh/zen as grok-4.6 list$. Not Pareto — nobody wins every named axis.spine SVG · in-house SVG
Is this comparison fair?
Yes, on the axes we set. Same session, same model, same seat:
zig-out/bin/graff(~127M) after hardlink pin + detachedlearn init. Six sandboxes that crossed the 5-call threshold leftgraff-pinnedon inode 1480190, nlink=7, same inode as the live exe. No extra 127M copy. Tasks with fewer than 5 API calls have no pin.opencode run --dir {sandbox}(Popen cwd is not enough).Prior rows on this PR paid a 132MB byte-copy (nlink=1) and/or ran OpenCode without
--dir. Those stay below as prior.Spine (exact-reply + file-ops + fix-fib) ·
run-20260830-101150-p)--dir)Not Pareto. OpenCode is faster to first token and to done. Graff is cheaper / fewer calls / smaller RSS.
In-house 6-task (
run-20260830-101332)--dir)Not Pareto. OpenCode is the only 6/6 — it spent 107.6s / 6 calls on
atomic-symlink-write. graff-dev missed that task in 3.53s / 1 call (captured answer was turn-pulse chrome; tests still fail). grok hit the real 240s budget with pipes still open (no usage footer; that row's list$ is $0 and undercounts grok spend).Prior (not this comparison)
Copy-tax graff + first OpenCode wiring, then a later
--dir-only OpenCode rerun:053939/--dir095006054427/--dir095035Hardlink cut graff spine wall 62.4s → 28.7s and in-house 361.1s → 80.8s. OpenCode
--diris required for a pass that the check can see.Blockers
atomic-symlink-write: oneshot chrome leaked as the answer; 1-call miss. Not an auth fail.atomic-symlink-write: real timeout (budget ended, pipes still open). Same miss as the prior in-house run.What this PR adds
opencode(--dir {sandbox}),opencode-zen,dsh-grok,dsh-xai--suite inhouse+ six fixtures + held-out checksplot_frontier.py(SVG)xai/grok-4.6; unknown models do not get fake $0learn initon this treePR #684 stays the learn-pin / hillclimb narrative. This PR is the multi-harness eval + fair A/B numbers. Do not merge to main from this agent.