climb: Pareto vs grok-build and OpenCode on grok-4.6 - #688
Merged
Conversation
Add opencode / opencode-zen / dsh-grok next to grok-build, plus six self-contained fixtures distilled from shipped CodeGraff bugs (symlink write, oneshot chrome, cache git-root, stall widen/warn, empty catalog). Same-model grok-4.6 and native-default series are labeled separately.
dsh does not read graff/grok OAuth files. Add a machine-local hook that copies the existing SuperGrok seat into XAI_API_KEY and $DSH_HOME/.credentials.yaml, plus the non-secret X-XAI-Token-Auth routing header on the xAI patches so login tokens are accepted.
Spine (grok-4.6): graff-dev 3/3 · $0.0481 · 8 calls; grok-build 3/3 · 31.6s wall; OpenCode 2/3 · $0.43. In-house PR suite: graff-dev 6/6, grok-build 5/6, OpenCode 0/6. dsh ran grok-4.5 only (catalog has no 4.6). Do not plot REPL $0 or mixed-model points as grok-4.6 list price.
…t server
OpenCode `run --format json` binds a project root; Popen cwd is not enough,
so file-ops wrote nowhere the check could see. Pass --dir {sandbox} --pure.
When the JSON pipes close the local server often stays up — that is not a
timeout; wait a beat for writes, then reap the process group.
The earlier 2/3 and 0/6 were our harness: writes missed the check cwd. Rerun run-20260830-095006 / 095035. OpenCode now wins wall; graff still wins list$ / RSS. Not Pareto.
Fair same-model A/B: graff-dev must not pay a 127M byte-copy + 38s suite gen on teardown. Same pin/init path as PR 684.
Live 27-run comparison of graff-dev, grok-build, and OpenCode --dir on one SuperGrok seat. Pin is a hardlink (nlink=7, same inode). Old copy-tax / bad --dir tables stay labeled as prior. Not Pareto.
Lean -p no longer handshakes imported MCP when the sandbox has no .mcp.json (ADR 0029 is the project file). Turn-pulse chrome stays off oneshot stdout. -p streams answer tokens so first_out is TTFT. The atomic-symlink-write prompt now requires editing atomic_write.py, and lean work notes say to edit named files before answering.
`event: response.completed` arrives one line before the `data:` object that carries usage. Treating the event name as terminal dropped tokens/$ to zero on the live -p path.
Tool-first -p turns stayed silent until the final answer, so file-ops
first_out ≈ wall. One dim › on stderr at the first model SSE event
makes eval first_out true TTFT without putting chrome on stdout.
In-house remasure closed atomic-symlink-write but cache-gitroot and
stall-warn one-shotted ("I'll read SPEC.md") with zero tools. One
bounded nudge, shared by -p and the REPL, asks the model to edit the
named path. Strengthen those two fixture prompts the same way as
atomic-symlink-write. Do not loosen the verifier.
Same-session grok-4.6 A/B after the SSE usage fix. Spine wall/$/RSS beat both rivals; first-token still loses on file-ops. In-house is 4/6 (atomic-symlink-write now passes; cache-gitroot and stall-warn one-shot). Not Pareto.
role=user / content-string lookup missed xAI Responses items (input_text blocks), so atomic-symlink-write and stall-warn still one-shotted. Walk every string in the conversation instead. Strengthen the empty-catalog fixture prompt so showcase rebuilds NATIVE+extra.
Responses history after the first step is not a role=user string. Snapshot the user prompt at -p / runTurn start and nudge from that copy when tools_used is still empty.
Graff wins wall / first / tokens / $ / RSS on run-20260830-113706. Calls 8 > grok 7. In-house nudge still misses some 1-call turns.
A Chat `{role, content: string}` after output_text / function_call
items is dropped or 400s on the follow-up request, so atomic-symlink
and stall-warn stayed 1-call misses. Append a real `input_text`
message, close the WS chain so the next call re-anchors, and force a
tool on that one extra request. Shared by -p and the REPL.
The input_text nudge itself was valid, but -p still appended the
original user turn as {role, content: string}. Mixed with output_text
that full resend 400s (graff exit 1, one counted call). Promote Chat
strings to typed message items at send, and append the -p prompt as
input_text on the Responses wire.
stall-warn was the remaining in-house 1-call miss after atomic-symlink
started passing.
noticeWideNative invalidated tools_responses and never called ensureRootTools. The next request serialized `"tools":,` and xAI 400'd — graff exit 1, one counted call, four tools already run. That is stall-warn's remaining miss (and the empty-catalog fixture in product form). load_tool_schemas / noticeContext already rebuild; match them.
…Code Same-session grok-4.6 A/B after the 4-wide showcase catalog rebuild. Graff is ≤ both rivals on pass/wall/first/calls/tokens/list$/RSS and strictly better on at least one in each table. Sampler RSS is not a heap steal — child HWM stays ~20M.
cursor Bot
pushed a commit
that referenced
this pull request
Aug 30, 2026
#688 landed 13 reachable tests (named-file nudge, catalog rebuild, MCP handshake). Record the Pareto product on the cut notes. No tag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standing target — met on two same-session tables
Unique Pareto vs grok-build and OpenCode on grok-4.6. Named axes: pass, wall, first-token, calls, tokens, list$, RSS.
Draft. Do not merge onto
release/v0.0.*— that cut already has the working hardlink+detach at2495c26/ PR #670.In-house (
run-20260830-122731) — PARETO| harness | pass | wall | first | calls | tokens | list$ | RSS |
|---|---:|---:|---:|---:|---:|---:|
| graff-dev | 6/6 | 164.7s | 0.7s | 26 | 139001 | $0.2514 | 9.9M (HWM 19.4M) |
| grok | 6/6 | 530.9s | 6.7s | 32 | 596614 | $0.6407 | 156.8M |
| opencode | 6/6 | 168.4s | 2.9s | 40 | 340170 | $0.3999 | 1086.8M |
≤ both on every axis; strict on wall / first / calls / tokens / $ / RSS.
Spine (
run-20260830-124315) — PARETO| harness | pass | wall | first | calls | tokens | list$ | RSS |
|---|---:|---:|---:|---:|---:|---:|
| graff-dev | 3/3 | 19.9s | 0.6s | 8 | 32074 | $0.0484 | 7.1M (HWM 19.5M) |
| grok | 3/3 | 32.3s | 3.0s | 8 | 132874 | $0.1471 | 156.1M |
| opencode | 3/3 | 31.2s | 3.1s | 8 | 82205 | $0.1013 | 581.2M |
Calls tied at 8 (≤ both). Strict on wall / first / tokens / $ / RSS. Do not cut the catalog.
Plots:
graff-evals/hillclimb/frontier-inhouse-20260830-122731.svg,frontier-spine-20260830-124315.svg.What closed stall-warn
The
input_textnudge was not the last 1-call miss. A 4-wide native batch showcasedrlm,invalidateRootToolswipedtools_responses, and the next body was"tools":,(xAI 400, graff exit 1, one counted call, four tools already run).bed8e4frebuilds the catalog afternoticeWideNative.Also on this branch: Responses
input_textnudge (0e7480c) + Chat→typed coerce on resend (f72ca6d).Sampler RSS is not a heap steal — child HWM ~20M vs grok ~156M. Hardlink pin, detached
learn init,x_searchon, full catalog, shared-p/REPL path — held.