feat(bench): standardized Wright Agent Score for model comparison - #491

Merged

Teakowa merged 32 commits into

mainfrom

Oct 4, 2026

Commits on Oct 4, 2026

feat(bench): add harness-free direct adapter and the evaluate command
Teakowa
committed
1dac661
feat(bench): add opencode and grok adapters and report the agent setup actually observed
Teakowa
committed
89e9659
fix(bench): classify a Devin catalog outage as a provider failure, back off retries, and grade a missing entry as an agent failure
Teakowa
committed
ef1ec04
feat(bench): hide answer keys and host checkouts from the agent with the file sandbox
Teakowa
committed
1b579b7
feat(bench): make evaluate runnable by hand with user defaults, a preflight, and a dry run
Teakowa
committed
605594e
feat(bench): compare score cards from separate evaluation runs
Teakowa
committed
35e30e6
fix(bench): hide withheld Devin tools from the agent and deny reading sibling runs and harness data
Teakowa
committed
bd7755a
fix(bench): clear adapter homes between infrastructure retries
Teakowa
committed
b430e29
feat(bench): allow-list file reads, record Devin effort from the model id, run sequentially by default, and write refreshed logins back
Teakowa
committed
2b014fc
feat(bench): add the suite command, a publishable results page, and a shell how-to
Teakowa
committed
24f187f
refactor(bench): keep runs, wikis, and pinned skills under one directory
Teakowa
committed
42d60ac
fix(bench): retry provider-interrupted trials when a run is repeated
Teakowa
committed
7e688be
fix(bench): refuse to continue a run with a different Wright binary
Teakowa
committed
338687b
fix(bench): harden score edge cases and cut adapter duplication
Teakowa
committed
4b7d548
fix(bench): keep agent text out of the provider-failure classification
Teakowa
committed
13cf1fe
fix(bench): finish hardening adapter stream parsing and retry cleanup
Teakowa
committed
0112e6d
test(bench): make the preflight count assertion host-independent
Teakowa
committed
6b45345
fix(bench): treat an Antigravity dropped connection as a provider failure and read the effort from the model id
Teakowa
committed
16e434f
fix(bench): serialize evaluate's effective options so its matrix.json reproduces the run
Teakowa
committed
7d00b00
fix(bench): refuse a score card when a run directory mixes read/write/network enforcement
Teakowa
committed
c5117cc
fix(bench): create the credential sync temp file at 0o600 and flag the claude-code sandbox login note
Teakowa
committed
404a075
fix(bench): resolve user path options when evaluate serializes its matrix
Teakowa
committed
16ebe2a
fix(bench): compare score identities with missing fields as 'not recorded'
Teakowa
committed
fee362e
fix(bench): close the remaining path and name gaps in manifest options
Teakowa
committed
e884ee5
fix(bench): classify malformed provider payloads and unblock adapter stdin
Teakowa
committed
a78c18d
fix(bench): hide the grading oracle outside overpy cells and the sandbox profile everywhere
Teakowa
committed
fd3e45f
fix(bench): match --only on the adapter alone or adapter:model exactly
Teakowa
committed
90e6377
fix(bench): drain adapter stderr concurrently with stdout
Teakowa
committed
3ca0c27
fix(bench): lock the sandbox profile so the agent cannot read it under another name
Teakowa
committed
778c827
fix(bench): let trial cleanup remove files the agent locked or hid
Teakowa
committed
e87deac
fix(bench): keep cleanup repairs on the run-dir node and retry by pass
Teakowa
committed
9e29351
fix(bench): harden run startup and resume against reviewer findings
Teakowa
committed
6d3b205