Run Pi on SuperGrok the same way as graff-dev SWE - #667
Closed
justrach wants to merge 2 commits into
Closed
Conversation
Add a pi-xai harness that maps graff login xai into XAI_API_KEY so `--suite swe --harness graff-dev,pi-xai --model grok-4.6` is the same seat A/B ADR 0024 used for grok-build. Stock Pi xai is an API-key provider; oauth type is ignored.
Same-login A/B (run-20260828-164346): Pi 5/6 in 276s / 166M vs graff 4/6 in 456s / 91M. Extra pass is json-stream whitespace-only json-seq (already named in ADR 0024). Do not steal Pi's catalog or heap.
justrach
marked this pull request as ready for review
August 28, 2026 16:50
Owner
Author
|
Already on main via the 281 cut (#670 / ADR 0043). Closing as superseded. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADR 0024 compared graff vs grok-build on
--suite swe. Pi was already wired aspi(OpenAI/luna) andpi-codegraff(Gemini gateway) — not the SuperGrok seat those numbers used.This adds
pi-xai: a wrapper that mapsgraff login xai(~/.xai/credentials/graff-oauth.json) intoXAI_API_KEYso stock Pi spends the same subscription. Stock Pi xai is an API-key provider;type=oauthis ignored.Live A/B (2026-08-28,
run-20260828-164346.jsonl)Same 6 DeepSWE-shaped tasks, grok-4.6, SuperGrok,
-j 6, one rep. Pi on stock defaults (4-tool catalog, thinking on).Pi's extra pass is
json-streamwhitespace-only json-seq — the same spec-contract miss ADR 0024 already named (grok-build passed it). Both misslabel-sort. Tokens and calls are a wash.Caveats: Pi
first_out_sis the first JSONLsessionline, not TUI first paint. Pi$is list-price; graff SuperGrok is flat-rate$0.0000.Reject: Pi's 4-tool catalog, Pi's 165M heap, treating
first_outas a TUI steal.Next speed cut (not this PR): every graff-dev SWE sandbox wrote 127M
.graff/learn-kit/graff-pinned(a copy of the binary). Pi's sandbox is ~8K. Skip auto-init on-pif we want the next wall/RSS drop. TUI leftover after the early alt-screen claim is first chrome, not another escape.Recorded as ADR 0043. 0042 is the TUI claim on #666.