In-house evals: six more shipped-PR verifiers - #690
Merged
Conversation
Double --suite inhouse with distilled verifiers from merged CodeGraff PRs: hardlink pin (#687), hosted x_search splice (#632), first-turn MCP skip (ADR 0035/0029), rlm showcase gate (#633), codedb five-shot menu (#597), peer-channel resume (#584). Same pattern as the first six — buggy starter + SPEC + visible tests + hidden/ check. Not the live repo.
Same-session grok-4.6 vs grok-build and OpenCode on the doubled PR suite (run-20260830-141658). Graff 192s / 52 calls / $0.350 vs OpenCode 236s / 74 / $0.792 and grok 462s / 63 / $1.115. First-token 0.02s is stderr › at boot — not scored as TTFT.
Visible README is ~200 lines (was ~1700). Same-model in-house 12/12 table replaces the retracted oneshot-learn-skip blurb. CLI / slash / providers stay in one <details>. Broken section links in embedding, ACP registry, and benchmarks/README follow the new headings.
cursor Bot
pushed a commit
that referenced
this pull request
Aug 31, 2026
Tip is hardlink + detached init, not the retracted oneshot skip. Point at the 12-task in-house table (#690).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Doubles
--suite inhouseand shortens the root README. Landed onrelease/v0.0.281(831272d). No tag. No merge to main.README
Visible page is ~200 lines (was ~1700). Same-model 12-task table is on it. CLI / slash / providers live in one
<details>. The retracted oneshot-learn-skip blurb is gone.Same-session remasure (
run-20260830-141658)Unique frontier on pass / wall / calls / tokens / list$ / RSS. Graff first-token 0.02s is boot stderr — not scored.