Hillclimb graff vs grok-build on grok-4.6 list-price axes

[justrach]

Score evals at xAI grok-4.6 list price, then copy what grok-build actually does — not a side bet.

The fix (shared -p / REPL path)

The 127M tax was learn_auto byte-copying zig-out/bin/graff into .graff/learn-kit/graff-pinned after five API calls (~38s CPU on fix-fib). grok-build does not do that.

Skipping learn-auto on -p hid the copy from oneshots and left the REPL/TUI still doing it. That path split is retracted.

This revision:

1. Hardlink graff-pinned (same inode, O(1); graff update replacing the live path leaves the pin). Copy only when the filesystem cannot. Proven at /tmp/graff-pin-proof: pin and zig-out/bin/graff share inode 1478920, nlink=2.
2. -p, piped graff repl, and TTY graff tui share one learn-auto path. Scripted graff repl prints the same [usage] footer so both surfaces score.
3. Session-end learn init is detached (ADR 0045) — same process-group spawn as a background trial. Suite generation is still ~38s of CPU; teardown no longer waits on it.

x_search stays on (ADR 0031). grok-build 1.0.5 still lists web_search / web_fetch. Heap and catalog untouched (ADR 0024).

Live A/B — run-20260830-042417.jsonl

Same SuperGrok / grok OAuth, grok-4.6, one rep, three core tasks. x_search on. -p and scripted graff repl were measured the same way.

harness	pass	wall	first	calls	tokens	list$	RSS
graff-dev (-p)	3/3	22.2s	2.8s	8	~32k	$0.0374	90.7M
grok-build	3/3	37.0s	3.9s	7	~116k	$0.1474	155.4M
graff-dev-repl	3/3	25.2s	0.03s (echo)	9	~64k	$0.1185	91.8M

REPL is fatter than -p because it is not lean-oneshot. REPL first_out_s is composer echo — ignore it; it is not time-to-first-model-token.

task	graff -p wall / $ / calls	grok wall / $ / calls
exact-reply	2.55s / $0.0067 / 1	2.68s / $0.0322 / 1
fix-fib	10.63s / $0.0170 / 4	26.90s / $0.0708 / 4
file-ops	8.99s / $0.0137 / 3	7.39s / $0.0444 / 2

This run's fix-fib used 4 API calls, so learn-auto did not fire (threshold is 5). No pin in those sandboxes.

Interactive TUI (same fix-fib)

graff tui --yolo --model grok-4.6 on a PTY (Ctrl+Q): test printed OK at 5.9s, turn 7.4s, session 9.3s including boot/quit. .graff was 52K. No graff-pinned, no extra 127M inode. Four API-call traces (quit during the fourth after the test passed), so learn-auto did not fire — same threshold miss as the -p row. First model token 2.6s.

Not strict Pareto

No. Grok-build still wins calls (7 vs 8 on -p, 7 vs 9 on REPL) and file-ops wall. We win wall, tokens, list$, and RSS. Pass is a tie. Do not claim a 5-axis win. Heap still theirs (155M vs 91M) — not stolen.

On this 3-point set both -p and grok-build are non-dominated (they own calls; we own the rest). That is a shared front, not a win.

Frontier graph

Two 2D projections of that live 3-task set (lower-left is better). Source: graff-evals/hillclimb/frontier-20260830.svg and baseline.md.

point	5-axis	wall vs $	calls vs $
graff-dev (-p)	on (cheaper / faster)	on (best both)	on (cheaper)
grok-build	on (fewer calls)	interior	on (fewer calls)
graff-dev-repl	interior (dominated by -p)	interior	interior

flowchart LR
  subgraph front[5-axis frontier - neither dominates]
    P[graff-dev -p: 22.2s / $0.0374 / 8 calls / on via cheaper and faster]
    G[grok-build: 37.0s / $0.1474 / 7 calls / on via fewer calls]
  end
  R[graff-dev-repl: 25.2s / $0.1185 / 9 calls / interior, dominated by -p]
  P -->|wins wall, calls, tokens, list$, RSS| R

Loading

 

[justrach]

Hillclimb product/evals landed via 281 (#687/#688/#690). Held items (oneshot skip, x_search off) stay out of tip. Closing as superseded.