Skip to content

Contract: a conditional sibling-regression allowance per benchmark - #468

Merged
renmengye merged 1 commit into
mainfrom
feat/conditional-sibling-regression
Oct 3, 2026
Merged

renmengye merged 1 commit into
mainfrom
feat/conditional-sibling-regression

Conversation

@renmengye

Copy link
Copy Markdown
Member

What

An optional per-benchmark regression block. Today a benchmark's single floor sets both its win bar and the regression it tolerates as a sibling. The block separates the two:

- name: rollout-mem
  direction: min
  min_delta_rel: 0.05         # a memory win
  regression:
    free_rel: 0.005           # up to 0.5% worse: always allowed
    max_rel: 0.5              # 50% worse or more: always refused
    requires_gain_rel: 0.05   # in between: only if the climbed benchmark gained at least 5%
  • Relative or absolute units, matching min_delta_rel / min_delta. An omitted free tolerance inherits the benchmark's floor, and the hard cap wins at equality.
  • Benchmarks without the block keep the original comparison, so every existing contract gates as before.
  • Verdict rows, run reports and PR tables name the rule that decided.

Motivating target: jepa.cpp, where a 27% GN-planner speedup cost 33% more memory, a trade the owner accepts. A single 5% memory floor would refuse it.

Verified

  • uv run pytest: 3,138 passed, 12 skipped. Ruff, mypy and gitleaks are clean.
  • The serial tier fails locally only because the sandbox denies ps.
  • contract_cli validates a contract that uses the block.

Review

  • Built by Codex.
  • Reviewed by Claude.
  • For the owner's review; not to be merged without it.

An optional regression block lets a benchmark tolerate a free regression,
cap it hard, and unlock the range in between only when the climbed
benchmark improves by a required gain. One floor set both a benchmark's win
bar and the regression it allowed on others, which could not express a
memory benchmark that should climb at 5% yet allow a large speed win to
cost some memory. Benchmarks without the block gate exactly as before;
verdicts name the rule that decided.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 — reviewed head 7489cf52 — reviewer summarizer:hermes/gpt-5.6-terra over coverage+credentials+deployment+general+lifecycle+prose.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

No findings were submitted by any review lens. Rejected findings: none.

@renmengye
renmengye merged commit 6ec24ea into main Oct 3, 2026
16 checks passed
@renmengye
renmengye deleted the feat/conditional-sibling-regression branch October 3, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant