Skip to content

Make task qualification count-rate two-sided - #158

Merged
MongLong0214 merged 1 commit into
devfrom
feat-issue-109
Jul 29, 2026
Merged

Make task qualification count-rate two-sided#158
MongLong0214 merged 1 commit into
devfrom
feat-issue-109

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

Closes #109.

Implements the inclusive 4/6–5/6 comparator count-rate gate, retains task-level exclusions with their measured rates, and removes excluded tasks from the analysis set.

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 1 commit in origin/dev..4d3bab848b43cdab9cc648b02b33220291804781
Active constraints: 11 limits · 21 ruled-out · 9 warnings — from 15 records over 4 changed paths

Active constraints for the paths this PR touches

Limits (11)

  • r-qual109 4d3bab8 — the existing M4 qualification rows do not carry reproposal_matches, so they cannot be requalified as count outcomes
  • r-rejwork141 5f7af65 — the counts read the diff, so an agent that pursued a rejected approach entirely in conversation and committed nothing registers zero
  • r-m5outcome 37f3400 — variance is qualified against a pilot, so an outcome with spread in the pilot and none in the study is still possible; this catches the registration error, not every future one
  • r-m4expose df84615 — exposure is observed through the shim's own append, so a harness that never installs the shim is caught by the absent-field refusal rather than by this test
  • r-protoreg31 1bd02c1 — the calibration arm cannot be used to select confirmatory tasks, only to size them
  • r-exposure52 ba69411 — legacy JSONL artifacts predate model and guard-exposure fields | their absence remains unknown and is never inferred or backfilled
  • r-4c73a2 1d82a5bcode inherits the limit of any lexical surface — an alternative implemented under another name is invisible, one named in a string literal is counted; both were true of artifacts
  • r-9c2f74 d653153 — the ablation arms cannot discriminate on these fixtures -- no-grade and no-lifecycle are byte-identical to the treatment in 9 of 10 tasks, because the seeds carry one reconstructed record and one task with a lifecycle trailer between them
  • r-9c2f74 d653153 — the harness assembles its own projection rather than calling the shipped injector, so what is measured is the harness's rendering of the records, not src/core/inject.ts (issue B-08 · Replace the benchmark harness injector with the actual src/core/inject.ts #36)
  • r-5c8b31 60ddc39 — the agent CLI exposes no in-flight turn limit, so a per-task turn budget can only ever be observed with this driver
  • r-6c2b95 5da793c — the CLI holds no API key, so any real driver runs through the user's own agent session and cannot be exercised in this environment

Ruled out (21)

  • r-qual109 4d3bab8 — treating absent comparator qualification as exclusion | it would hide a missing measurement rather than record a measured rate
  • r-rejwork141 5f7af65 — a single weighted rejected-work score | the weights would carry the argument, and a reader whose situation weights dependency additions above hunks cannot recover the components from a composite
  • r-m5outcome 37f3400 — reusing matched.length as the count | it equals the re-proposal count only while violations stays zero, so it would measure correctly for the wrong reason and stop the day the other field was fixed
  • r-m4expose df84615 — asserting on guard_exposure being present | the M4 failure is exactly a field that looks right while the code behind it does nothing
  • r-protoreg31 1bd02c1 — keeping GEE with a small-sample correction | permutation is exact rather than corrected, and the correction literature stops short of an ICC this high
  • r-protoreg31 1bd02c1 — a per-task qualification threshold read off outcomes | any such threshold reintroduces the selection that saturated M4
  • r-protoreg31 1bd02c1 — leaving the duplicate id and exempting it in the test | the check is right and the history was wrong
  • r-exposure52 ba69411 — compute a treatment effect with unknown guard exposure | an old row that never recorded whether the guard reached the run cannot distinguish no treatment from an ignored treatment
  • r-8e15c2 70f6bcf — hashing only the files the entry point transitively imports | it is the same judgement call that produced this bug, made with more machinery
  • r-4a17d9 68d4c92 — guard versus the control as the primary test | it would confound the route with whether records exist at all, and the interesting question is which route works better once they do
  • r-4a17d9 68d4c92 — a blocking guard arm | GUARD-CANNOT-BLOCK measured true and false positives in one score band; the only precision-safe threshold catches one in five
  • r-4c73a2 1d82a5b — re-scoring the recorded transcripts and reporting the result | the labels would be chosen with the outcome known, which §4 forbids; the recorded runs were used only to check the surface behaves as intended
  • r-4c73a2 1d82a5b — moving violation_if to code as well | it is instrumented-only, and changing an unregistered metric alongside a registered one makes it unclear which change any difference belongs to
  • r-6d92c4 b058de2 — documenting grep -a as the convention | it leaves every future reader one forgotten flag away from the same silent empty result, and the cause takes six characters to remove
  • r-9c2f74 d653153 — resume the pilot into the same file | a new process would load the edited code and create the mixing that had not happened
  • r-9c2f74 d653153 — run the ablation arms as they stand | three nulls from comparing identical inputs read as "these guarantees do not matter"
  • r-9c2f74 d653153 — keep the tasks that showed an effect and rewrite only the rest | the property is the criterion, not the direction of the result
  • r-5c8b31 60ddc39 — keep turns and explain it in prose | the JSONL outlives the prose, and whoever reads the rows later will not have it
  • r-5c8b31 60ddc39 — drop the turn budget since it cannot be enforced | the overrun is still the signal that a run went off the rails
  • r-6c2b95 5da793c — skip the dry-run driver | then nothing exercises the harness until a key exists, and the first real run debugs the harness instead of measuring anything
  • r-6c2b95 5da793c — emit dry-run rows without a marker | indistinguishable from measurements the moment they leave the terminal

Warnings (9)

  • r-qual109 4d3bab8 (claim) — callers must pass the primary comparator's six calibration rows for every candidate task
  • r-exposure52 ba69411 (claim) — guard exposure is instrumented in the benchmark hook adapter, not by changing guard scoring | M4 rows remain unexposed and metrics refuses their effect estimate
  • r-8e15c2 70f6bcf (claim) — this is the second time in two commits that a fix for the M3 failure did not actually cover the M3 failure. The first was caught by testing the fix against the incident rather than against its own tests
  • r-4a17d9 68d4c92 (claim) — both hooked arms pay a subprocess per edit. M2 put all three of its timeouts in its hooked arm, and a timeout truncates a run, which flatters whichever arm suffers it — stated before the numbers exist
  • r-4c73a2 1d82a5b (claim) — SURFACES in task-loader.ts and MatchSurface in types.ts are two lists of the same thing — adding a surface means editing both
  • r-6d92c4 b058de2 (claim) — keep the NUL separator when touching these keys — a printable delimiter can appear inside a trailer value and would collide
  • r-9c2f74 d653153 (claim) — after the measurement, check that git status is clean and the recorded sha is still HEAD -- an edit mid-run breaks reproducibility silently, and that check is the only thing that catches it
  • r-5c8b31 60ddc39 (claim) — a driver that gains a real turn limit should stop emitting over-turns and start emitting an enforced label -- do not reuse over-turns for something the harness actually stopped
  • r-6c2b95 5da793c (claim) — never cite a row with simulated:true -- README numbers come from bench/results logs and those rows are not results

git log --follow accepts exactly one pathspec, so renames are not followed for 4 paths; query one path at a time to follow its rename chain

withheld the content of 1 record(s) graded blocked: a Ruled-out trailer matching an injection pattern is reported, never quoted (SPEC §7)

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

The two-sided count-rate gate must not silently discard a task whose
comparator calibration rows are absent.

Ruled-out: treating absent comparator qualification as exclusion | it would hide a missing measurement rather than record a measured rate
Limit: the existing M4 qualification rows do not carry reproposal_matches, so they cannot be requalified as count outcomes
Warn: callers must pass the primary comparator's six calibration rows for every candidate task
Blast: module
Undo: easy
Certainty: firm
Record-Id: r-qual109
@MongLong0214
MongLong0214 merged commit 87091ab into dev Jul 29, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make benchmark task qualification two-sided (4–5 of 6)

1 participant